Summarize prediction-interval validation

Calculates a set of complementary diagnostics for evaluating central prediction intervals. The function combines empirical coverage, departure from nominal coverage, interval width, and the interval score in a single one-row summary.

Usage

uncertainty_metrics(
  obs,
  lower = NULL,
  upper = NULL,
  level = 0.95,
  na.rm = TRUE,
  pred = NULL,
  predictive_sd = NULL,
  distribution = NULL
)

Arguments

obs: Numeric observation vector.
lower, upper: Numeric vectors containing the lower and upper prediction-interval bounds. In explicit-bound mode, both must be supplied and have the same length as obs.
level: Nominal central prediction-interval coverage probability, strictly between zero and one. The default is 0.95.
na.rm: Logical; remove incomplete observation/interval combinations? The default is TRUE.
pred, predictive_sd: Optional numeric vectors of predictive means and predictive standard deviations, supplied together and of the same length as obs. Assumes normal predictive distributions. Non-missing predictive standard deviations must be finite and strictly positive.
distribution: Optional numeric matrix or data frame of equally weighted predictive samples: one row per observation and one column per predictive draw. Supply this instead of bounds or predictive means and standard deviations. At least one draw is required; infinite values are not allowed.

Value

A one-row base data frame containing:

picp: Empirical prediction interval coverage probability.
picp_error: Difference between empirical and nominal coverage, calculated as picp - level.
interval_width: Mean prediction-interval width.
interval_score: Mean interval score at the specified nominal coverage level.

Details

Supply lower and upper, predictive mean and standard deviation (pred and predictive_sd), or predictive samples (distribution). For a nominal interval level \(p\), a well-calibrated uncertainty model should have empirical prediction interval coverage probability (PICP) close to \(p\).

The returned diagnostics describe complementary aspects of prediction- interval performance:

picp is the proportion of observations contained within the prediction intervals;
picp_error is empirical minus nominal coverage. Values below zero indicate under-coverage, whereas values above zero indicate over-coverage;
interval_width is the mean width of the prediction intervals and summarizes their sharpness. Smaller values indicate narrower intervals, but are desirable only when calibration remains adequate;
interval_score is a proper scoring rule that rewards narrow intervals while penalizing observations falling below or above them. Smaller values indicate better performance when comparing predictions on the same response scale and at the same nominal interval level.

These quantities should be interpreted together. Coverage alone does not reward sharp predictions, while interval width alone does not assess whether the intervals contain observations at the stated frequency. The interval score combines both aspects in a single statistic.

This function summarizes prediction intervals at one level. For calibration across multiple interval levels, use gg_coverage() and accuracy_plot_metrics(). For predictive quantiles, use qcp() and gg_qcp(). For complete predictive distributions, use pit(), gg_pit(), crps(), or crps_decomposition().

All four returned statistics use the same complete observation/lower/upper triplets. This differs intentionally from standalone interval_width(), which is observation-independent and can use interval bounds where obs is missing.

Input representations

Supply exactly one of explicit lower and upper bounds, predictive mean and standard deviation (pred and predictive_sd), or predictive samples (distribution). Normal inputs generate central intervals using normal quantiles. Samples generate equal-tailed intervals using stats::quantile() with type = 7, at probabilities (1 - level) / 2 and (1 + level) / 2. With na.rm = TRUE, a case is removed if its observation or any supplied predictive value is missing; individual missing draws are not discarded within a case. With na.rm = FALSE, incomplete inputs give NA. No complete cases also gives NA. Standalone interval_width() ignores missing obs.

References

Gneiting, T. and Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102, 359-378. doi:10.1198/016214506000001437

Schmidinger, J. and Heuvelink, G. B. M. (2023). Validation of uncertainty predictions in digital soil mapping. Geoderma, 437, 116585. doi:10.1016/j.geoderma.2023.116585

Examples

obs <- c(1, 2, 3, 4, 5)
lower <- c(0, 1, 2, 3, 4)
upper <- c(2, 3, 4, 5, 6)

uncertainty_metrics(
  obs,
  lower = lower,
  upper = upper,
  level = 0.80
)
#>   picp picp_error interval_width interval_score
#> 1    1        0.2              2              2
uncertainty_metrics(obs, pred = obs, predictive_sd = rep(1, 5), level = 0.8)
#>   picp picp_error interval_width interval_score
#> 1    1        0.2       2.563103       2.563103
uncertainty_metrics(obs, distribution = cbind(obs - 1, obs, obs + 1))
#>   picp picp_error interval_width interval_score
#> 1    1       0.05            1.9            1.9