Dataset Shift with Outlier Scores
Convert P-value to Bayes Factor
Convert Bayes Factor to P-value
Asymptotic Test from Outlier Scores
Asymptotic Test With Out-Of-Bag Scores
Bayesian and Frequentist Test from Outlier Scores
Bayesian Test from Outlier Scores
dsos: Dataset Shift with Outlier Scores
Plot Bayesian test for no adverse shift.
Plot frequentist test for no adverse shift.
Print Bayesian test for no adverse shift.
Print frequentist test for no adverse shift.
Permutation Test from Outlier Scores
Permutation Test With Out-Of-Bag Scores
Permutation Test By Refitting
Weighted AUC from Outlier Scores
Test for no adverse shift in two-sample comparison when we have a training set, the reference distribution, and a test set. The approach is flexible and relies on a robust and powerful test statistic, the weighted AUC. Technical details are in Kamulete, V. M. (2021) <arXiv:1908.04000>. Modern notions of outlyingness such as trust scores and prediction uncertainty can be used as the underlying scores for example.