Summarise Performance of a Prediction Model

Standardised reporting of binary prediction model performance. Returns a data frame of metrics (point estimate + confidence interval) and optionally renders it via reportRmd::outTable().

predsum(
  true,
  predicted,
  cutoff = NULL,
  positive = NULL,
  ci = c("clopper-pearson", "bootstrap"),
  ci_width = 0.95,
  ci_separate = FALSE,
  nboot = 2000,
  seed = 1,
  digits = 2,
  output = "epi",
  tableOnly = FALSE
)

Arguments

true: Vector of true class labels. Must be 0/1 or a two-level factor.
predicted: Vector of predicted values, either dichotomous (0/1) or numeric scores. If scores, cutoff must be supplied.
cutoff: Numeric cut-off applied as predicted >= cutoff to define the positive class. Required when predicted is non-binary and ignored (with a message) otherwise.
positive: The value of true treated as the positive class. Defaults to 1 when true is 0/1, or to the second level when true is a factor.
ci: Confidence interval method: "clopper-pearson" (default; Delong is used for AUC) or "bootstrap" (percentile).
ci_width: Nominal coverage, default 0.95.
ci_separate: logical. If FALSE (default) estimate and CI are combined in a single Value (95%CI) column. If TRUE, the output has four columns: Statistic, Estimate, lower, upper.
nboot: Number of bootstrap replicates, default 2000.
seed: integer seed set for the duration of the bootstrap so that confidence intervals are reproducible; the calling session's random state is restored afterwards. Defaults to 1. Set to NULL to leave the random state untouched, in which case intervals will vary between calls.
digits: Number of digits for rounding, default 2.
output: One of "epi" (default), "data-science", "all", or a character vector of statistic names. Recognised names (case-insensitive): prevalence, sensitivity, specificity, ppv, npv, accuracy, auc, aucpr, f1, brier, tp, fp, tn, fn, n, events, positive, cutoff. The aliases recall and precision are accepted and, when supplied, are reflected in the row labels. Preset names cannot be combined with individual statistics.
tableOnly: logical. If TRUE the data frame is returned silently; if FALSE (default) the data frame is rendered via reportRmd::outTable() and returned invisibly.

Value

A data frame of performance statistics.

Details

Clopper-Pearson intervals are used for proportion-based metrics (prevalence, sensitivity, specificity, PPV, NPV, accuracy). AUC uses the Delong variance estimator on the logit scale. Brier score, F1 and AUC-PR have no closed-form interval and are obtained by percentile bootstrap; when ci = "clopper-pearson" this is done automatically and a message is emitted.

When predicted is dichotomous, AUC, AUC-PR, Brier and the cut-off row are omitted.

References

Delong ER, Delong DM, Clarke-Pearson DL (1988). Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44(3):837-845.

Clopper CJ, Pearson ES (1934). The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26(4):404-413.

Examples

# Simulate true labels and model scores
set.seed(1)
n <- 300
true  <- rbinom(n, 1, 0.3)
score <- plogis(rnorm(n, mean = ifelse(true == 1, 0.9, -0.9)))

# Epidemiology-style summary from probability scores
predsum(true, score, cutoff = 0.5, tableOnly = TRUE)
#> Bootstrapping CIs for Brier, F1 and AUC-PR (2000 replicates, seed = 1).
#>            Statistic     Value (95%CI)
#> 1         Prevalence 0.28 (0.23, 0.33)
#> 2        Sensitivity 0.86 (0.76, 0.92)
#> 3        Specificity 0.80 (0.74, 0.85)
#> 4                PPV 0.62 (0.52, 0.71)
#> 5                NPV 0.94 (0.89, 0.97)
#> 6           Accuracy 0.81 (0.76, 0.86)
#> 7                AUC 0.89 (0.84, 0.93)
#> 8        Sample Size               300
#> 9             Events                83
#> 10 Positive Category                 1
#> 11           Cut-off               0.5

# Data-science style (Recall, Precision, F1, Brier, AUC-PR)
predsum(true, score, cutoff = 0.5,
        output = "data-science", tableOnly = TRUE)
#> Bootstrapping CIs for Brier, F1 and AUC-PR (2000 replicates, seed = 1).
#>            Statistic     Value (95%CI)
#> 1         Prevalence 0.28 (0.23, 0.33)
#> 2             Recall 0.86 (0.76, 0.92)
#> 3        Specificity 0.80 (0.74, 0.85)
#> 4          Precision 0.62 (0.52, 0.71)
#> 5                NPV 0.94 (0.89, 0.97)
#> 6           Accuracy 0.81 (0.76, 0.86)
#> 7             AUC-PR 0.80 (0.71, 0.87)
#> 8                 F1 0.72 (0.64, 0.78)
#> 9              Brier 0.14 (0.13, 0.16)
#> 10       Sample Size               300
#> 11            Events                83
#> 12 Positive Category                 1
#> 13           Cut-off               0.5

# Custom selection with separate CI columns
predsum(true, score, cutoff = 0.5,
        output = c("sensitivity", "specificity", "auc"),
        ci_separate = TRUE, tableOnly = TRUE)
#> Bootstrapping CIs for Brier, F1 and AUC-PR (2000 replicates, seed = 1).
#>     Statistic Estimate lower upper
#> 1 Sensitivity     0.86  0.76  0.92
#> 2 Specificity     0.80  0.74  0.85
#> 3         AUC     0.89  0.84  0.93

# Dichotomous predictions: AUC, AUC-PR, Brier and cut-off are omitted
pred_class <- as.integer(score >= 0.5)
predsum(true, pred_class, tableOnly = TRUE)
#> Bootstrapping CIs for Brier, F1 and AUC-PR (2000 replicates, seed = 1).
#>           Statistic     Value (95%CI)
#> 1        Prevalence 0.28 (0.23, 0.33)
#> 2       Sensitivity 0.86 (0.76, 0.92)
#> 3       Specificity 0.80 (0.74, 0.85)
#> 4               PPV 0.62 (0.52, 0.71)
#> 5               NPV 0.94 (0.89, 0.97)
#> 6          Accuracy 0.81 (0.76, 0.86)
#> 7       Sample Size               300
#> 8            Events                83
#> 9 Positive Category                 1