Adventures in Model Validation: Why RMSE is NOT a Stand-Alone Measure
November 10th, 2014It seems like the major component of model validation is Root Mean Square Error, or RMSE. Â RMSE is basically this:
$$RMSE=\sqrt{\frac{\sum{(Count-Model)}^{2}}{N}}$$
And %RMSE is:
$$\%RMSE=\frac{RMSE}{\dfrac{\sum{Count}}{N}}*100 $$
These are useful measures to measure the error, but the problem with using them as a wholesale measure is that they ignore the DIFFERENCE of the error – this is evident in the numerator of the RMSE equation where the difference is squared. Any number squared becomes POSITIVE.
In the Model Validation and Reasonableness Checking Manual, the first item in assignment aggregate checks is VMT, as it well should be. Â Consider the following scenario:
Two model runs, one with assignments 20-40% high, and the other with assignments 20-40% low, both compared to the counts.  They can have nearly the same RMSE (overall, it’ll probably be around 30% FOR BOTH), but the VMT will show one ~30% high and the other ~30% low.