Summarize calibration from an accuracy plot

Computes numerical summaries of the departures from the 1:1 line in a prediction-interval reliability plot, also known in geostatistics as an accuracy plot. The approach evaluates prediction interval coverage probability (PICP) over multiple nominal interval levels.

Usage

accuracy_plot_metrics(
  obs,
  lower = NULL,
  upper = NULL,
  level = NULL,
  pred = NULL,
  predictive_sd = NULL,
  levels = NULL,
  na.rm = TRUE,
  distribution = NULL
)

Arguments

obs: Numeric observation vector.
lower, upper: Named lists of lower and upper prediction-interval bounds. Names must represent nominal coverage levels such as "0.50" or "0.95". A single pair of numeric vectors is also accepted when level is supplied.
level: Nominal central prediction-interval coverage for a single numeric lower/upper pair. It must be NULL when named lists are supplied.
pred, predictive_sd: Optional numeric vectors of predictive means and predictive standard deviations. When supplied together, central prediction intervals are generated assuming normal predictive distributions. Do not also supply lower or upper.
levels: Nominal central prediction-interval coverage probabilities used when pred and predictive_sd, or distribution, are supplied. Values must lie strictly between zero and one. Defaults to every percentage from 1% to 99%.
na.rm: Logical; remove incomplete observation/interval combinations? With interval lists, only cases complete in obs and both bounds at every supplied level are used, so all levels share the same validation sample. If FALSE, any incomplete case makes coverage missing at every level. With predictive samples, a row missing any draw or obs is incomplete.
distribution: Optional numeric matrix or data frame of equally weighted predictive samples: one row per observation and one column per predictive draw. Supply this instead of bounds or predictive means and standard deviations. At least one draw is required; infinite values are not allowed.

Value

A one-row data frame containing:

absolute_deviation: Total area between the empirical coverage curve and the 1:1 line. Zero is ideal.
over_uncertainty: Area above the 1:1 line, corresponding to over-coverage.
under_uncertainty: Area below the 1:1 line, corresponding to under-coverage.
over_percent: Percentage of total absolute deviation occurring above the 1:1 line.
under_percent: Percentage of total absolute deviation occurring below the 1:1 line.

Details

For a well-calibrated uncertainty model, the empirical coverage $\mathrm{PICP}(p)$ should be close to the nominal coverage probability $p$ over the range of evaluated prediction intervals. Perfect calibration therefore corresponds to the 1:1 line.

The overall departure from this line is summarized by the absolute area

$$ A = \int_0^1 |\mathrm{PICP}(p)-p|\,dp. $$

absolute_deviation is the exact area for the piecewise-linear interpolant of the evaluated coverage curve. Integration splits segments at crossings of the 1:1 line before applying the trapezoidal rule. A value of zero indicates perfect calibration; larger values indicate greater overall disagreement between nominal and empirical coverage.

The total deviation is further separated according to whether the empirical coverage curve lies above or below the 1:1 line:

$$ A_{\mathrm{over}} = \int_0^1 \max\{\mathrm{PICP}(p)-p,0\}\,dp, $$

$$ A_{\mathrm{under}} = \int_0^1 \max\{p-\mathrm{PICP}(p),0\}\,dp. $$

over_uncertainty is the area above the 1:1 line, where empirical coverage exceeds nominal coverage. This corresponds to over-coverage and is generally associated with prediction intervals that are too wide, or predictive uncertainty that is overestimated.

under_uncertainty is the area below the 1:1 line, where empirical coverage is smaller than nominal coverage. This corresponds to under-coverage and is generally associated with prediction intervals that are too narrow, or predictive uncertainty that is underestimated.

over_percent and under_percent give the relative contributions of these two components to the total absolute deviation. They describe percentages of the total area of miscalibration, not percentages of observations. When absolute_deviation is greater than zero, the two percentages sum to 100.

The integration is performed over the supplied nominal levels after adding the endpoints (0, 0) and (1, 1) to the reliability curve. For a normal predictive distribution, supplying pred and predictive_sd without levels evaluates the default sequence of nominal interval levels from 1% to 99%. The added endpoints are assumptions, not measured coverage values; the resulting areas depend on the supplied grid and this interpolation. Interval lists use cases complete across all levels, as in gg_coverage(). If coverage cannot be calculated, all five summaries are NA with a warning. When total deviation is zero, the two percentage contributions are NA because there is no deviation to apportion; this does not issue a warning.

These summaries describe calibration rather than sharpness. They should therefore be interpreted together with measures such as prediction interval width or a proper scoring rule when comparing predictive uncertainty.

Accuracy plots were proposed for the direct assessment of local uncertainty by Deutsch (1997) and subsequently applied to geostatistical uncertainty evaluation in soil science by Goovaerts (2001). Related numerical summaries of departure from the accuracy-plot reference line were used by Wadoux, Brus, and Heuvelink (2018).

Predictive samples can be supplied directly using distribution. Central empirical intervals are generated at levels as in gg_coverage(), using type-7 quantiles and the same complete cases at every level.

References

Deutsch, C. V. (1997). Direct assessment of local accuracy and precision. In E. Y. Baafi and N. A. Schofield (Eds.), Geostatistics Wollongong '96, pp. 115-125.

Goovaerts, P. (2001). Geostatistical modelling of uncertainty in soil science. Geoderma, 103, 3-26. doi:10.1016/S0016-7061(01)00067-2

Wadoux, A. M. J.-C., Brus, D. J. and Heuvelink, G. B. M. (2018). Accounting for non-stationary variance in geostatistical mapping of soil properties. Geoderma, 324, 138-147.

Examples

set.seed(123)
n <- 200
pred <- seq(0, 10, length.out = n)
predictive_sd <- rep(1, n)
obs <- stats::rnorm(n, mean = pred, sd = predictive_sd)

accuracy_plot_metrics(
  obs,
  pred = pred,
  predictive_sd = predictive_sd
)
#>   absolute_deviation over_uncertainty under_uncertainty over_percent
#> 1         0.02521667       0.02383333       0.001383333     94.51421
#>   under_percent
#> 1       5.48579
accuracy_plot_metrics(1:3, distribution = cbind(0:2, 1:3, 2:4),
                      levels = c(0.5, 0.9))
#>   absolute_deviation over_uncertainty under_uncertainty over_percent
#> 1               0.25             0.25                 0          100
#>   under_percent
#> 1             0