Calculate prediction-validation metrics for one or more models

Applies the individual prediction metrics to one or more prediction vectors. Each statistic is returned once. Efficiency is named R2; standalone nse() and mec() remain aliases. See the linked metric help pages for equations, references, and interpretation.

Usage

model_metrics(mods, obs, na.rm = TRUE, extended = FALSE, digits = NULL)

Arguments

mods: Numeric vector, list of numeric vectors, or numeric matrix/data frame with one model per column. Rows match obs in order. Supplied model names must be unique; missing names are generated.
obs: A numeric observation vector.
na.rm: Logical; whether incomplete observation-prediction pairs should be removed. The default is TRUE. If FALSE, incomplete pairs result in missing statistics rather than being silently removed.
extended: Logical; include additional robust, scale-normalised, percentage, agreement, and KGE (2009) metrics?
digits: Integer or NULL. If supplied, round numeric results to this many digits (0 to 22). NULL retains full numerical precision.

Value

A base data frame with one row per model and canonical columns model, bias, mae, mse, rmse, nrmse, crmse, correlation, r2, R2, sd_ratio, ccc, and Cb. With extended = TRUE, adds mdae, rpd, rpiq, sep, rer, mape, mpe, smape, msle, rmsle, rae, rrmse, willmott_d, and kge (KGE (2009)). Duplicate columns ME, MAE, RMSE, r, nse, NSE, MEC, and rhoC have been removed; use bias, mae, rmse, correlation, R2, and ccc instead.

Details

Errors are observation minus prediction. See bias(), crmse(), nse(), and ccc() for definitions and interpretation.

Errors are observation minus prediction: negative ME indicates overprediction. ME, MAE and RMSE have the input units. NSE is one minus the ratio of squared error to the observation sum of squares about its mean. Zero is the observation-mean benchmark and negative values are worse. Concordance uses population variances (divisor n); rhoC = r * Cb. Missing pairs are removed separately for each model, so comparisons may use different subsets. NA and NaN are missing; infinite values are rejected. Repeated observations are retained with equal weight. With fewer than two pairs, correlation and efficiency are NA. Constant inputs return NA for r and r2 because Pearson correlation is undefined. Constant observations give NA efficiency. If either vector is constant, Cb and rhoC are zero when the concordance denominator is positive; both are NA for identical constant vectors (zero denominator). All-missing models return NA.

Bias correction factor

Cb is Lin's bias correction factor, using population SDs and means: $$C_b = \frac{2\sigma_o\sigma_p}{\sigma_o^2+\sigma_p^2+(\mu_o-\mu_p)^2}.$$ It ranges from zero to one; one indicates equal means and SDs. It does not measure correlation. For defined correlation, CCC equals r times Cb. See ccc() and Lin (1989), doi:10.2307/2532051.

References

Wadoux, A. M. J.-C., Walvoort, D. J. J., and Brus, D. J. (2022). An integrated approach for the evaluation of quantitative soil maps through Taylor and solar diagrams. Geoderma, 405, 115332. doi:10.1016/j.geoderma.2021.115332

Examples

obs <- c(1, 2, 3, 4, 5)
preds <- list(model_a = c(1, 2, 3, 4, 5),
              model_b = c(2, 2, 3, 4, 4))
model_metrics(preds, obs)
#>           model         bias mae mse      rmse nrmse     crmse correlation  r2
#> model_a model_a  0.00000e+00 0.0 0.0 0.0000000   0.0 0.0000000   1.0000000 1.0
#> model_b model_b -5.55247e-17 0.4 0.4 0.6324555   0.4 0.6324555   0.9486833 0.9
#>          R2  sd_ratio       ccc        Cb
#> model_a 1.0 1.0000000 1.0000000 1.0000000
#> model_b 0.8 0.6324555 0.8571429 0.9035079