rmse function

(Normalized) root mean square error