statistics_calibratR function

statistics_calibratR

this method offers a variety of statistical evaluation methods for the output of the calibrate method. All returned error values represent mean error values over the n_seeds times repeated 10-fold CV.

statistics_calibratR(calibrate_object, t.test_partitions = TRUE,
  significance_models = TRUE)

Arguments

• calibrate_object: list that is returned from the calibrate function. The parameter n_seeds is available as a list component of the calibrate_object
• t.test_partitions: Performs a paired two sided t.test over the error values (ECE, CLE1, CLE0, MCE, AUC, sensitivity and specificity) from the random partition splits comparing a possible significant difference in mean among the calibration models. All models and the original, scaled and transformed values are tested against each other. The p_value and the effect size of the t.test are returned to the user. Can only be performed, if the calibrate_object contains a summary_CV list object, else, an error is returned. Default: TRUE
• significance_models: returns important characteristics of the implemented calibration models, Default: TRUE

Returns

An object of class list, with the following components: - mean_calibration: mean of calibration error values (ECE_equal_width, MCE_equal_width, ECE_equal_freq, MCE_equal_freq, RMSE, Class 1 CLE, Class 0 CLE, Brier Score, Class 1 Brier Score, Class 0 Brier Score) over n_seeds times repeated 10-fold CV. ECE and MCE are computed once using equal-width and once using equal-frequency binning for the construction of the underlying binning scheme. Only returned, if calibrate_object contains a summary_CV list object.

• standard_deviation: standard deviation of calibration error values over n_seeds times repeated 10-fold CV. Only returned, if calibrate_object contains a summary_CV list object.

• var_coeff_calibration: variation coefficient of calibration error values over n_seeds times repeated 10-fold CV. Only returned, if calibrate_object contains a summary_CV list object.

• mean_discrimination: mean of discrimination error (sensitivity, specificity, AUC, positive predictive value, negative predictive value, accuracy) values over n_seeds times repeated 10-fold CV. The "cut-off" is the cut-off value that maximizes sensitivity and specificity. Only returned, if calibrate_object contains a summary_CV list object.

• sd_discrimination: standard deviation of discrimination error values over n_seeds times repeated 10-fold CV. Only returned, if calibrate_object contains a summary_CV list object.

• var_coeff_discrimination: variation coefficient of discrimination error values over n_seeds times repeated 10-fold CV. Only returned, if calibrate_object contains a summary_CV list object.

• t.test_calibration: =list(p_value=t.test.calibration, effect_size=effect_size_calibration), only returned if t.test=TRUE

• t.test_discrimination: =list(p_value=t.test.discrimination, effect_size=effect_size_discrimination), only returned if t.test=TRUE

• significance_models: only returned if significance_models=TRUE

• n_seeds: number of random data set partitions into training and test set for folds-times CV

• original_values: list object that consists of the actual and predicted values of the original scores

Details

DETAILS

Examples

## Loading dataset in environment
 data(example)
 calibration_model <- example$calibration_model

 statistics <- statistics_calibratR(calibration_model)

See Also

t.test,friedman.test

Author(s)

Johanna Schwarz