# Predictive accuracy measurement using Bayesian Model Averaging This function is used for predictive accuracy measurement of the selected models using Bayesian Model Averaging methodology. The Occam's window with cut out threshold of `thr` is used. That means only models that have posterior probability of at least `thr` * posteior probability of the highest posterior probability model are considered in model averaging. For survival time response datasets, the predictive Area Under Curve (AUC) at each given time point is computed as the output. In this case, the predictive AUC is obtained using Uno's method for observations in the test set. For binary outcome data, only one AUC is reported which is from the ROC computed on the test set. The training set is used to find the selected model and relevant probabilities. ```r predBMA( bvsobj, X, resp, prep, logT, nlptype = "piMOM", train_idx, test_idx, thr = 0.05, times = NULL, family = c("logistic", "survival") ) ``` ## Arguments - `bvsobj`: An object that is generated by `bvs` function. It is the output of the Bayesian variable selection procedure. - `X`: The same `n` times `p` data frame initially used in `bvs` function. This is the full data frame containing all test and train data. - `resp`: For logistic regression models, this variable is the binary response vector. For the Cox proportional hazard models this is a two column matrix where the first column contains survival times and the second column is the censoring status for each observation. Note that for survival times, the time section of this variable should be in the same scale and unit (year, days, etc.) as `times` variable for which the AUC has to be computed. - `prep`: A boolean variable determining if the preprocessing step has been done on the original data frame in `bvs` function. This should have the same value as `prep` variable in the `bvs` function. - `logT`: A boolean variable determining if log transform should has been done on continuous columns of the data frame in `bvs` function. This should have the same value as `logT` variable in the `bvs` function. - `nlptype`: Determines the type of nonlocal prior that is used in the analyses. It can be "piMOM" for product inverse moment prior, or "pMOM" for product moment prior. The default is set to piMOM prior. - `train_idx`: An integer vector containing the indices of the training set. - `test_idx`: An integer vector containing the indices of the test set. The set of observations that prediction will be performed on. - `thr`: The threshold used for Occam's window as explained in the description. The default value for this variable is 0.05. - `times`: A vector of times at which predictive AUC is to be computed. This input is only used for prediction in survival data analysis. - `family`: Determines the type of data analysis. `logistic` is for binary outcome and logistic regression model whereas, `survival` represents survival outcomes and the Cox proportional hazard model. ## Returns The output is different based on the family for the anlysis of data 1) `family = logistic` The output is a list with the two following objects: - **auc**: This is the area under the ROC curve after Bayesian model averaging is used to obtain ROC for the test data. - **roc_curve**: This is a two column matrix representing points on the ROC curve and can be used to plot the curve. The first column is FPR and the second column is TPR which represent x-axis and y-axis in the ROC curve, respectively. 2) `family = survival` - **auc**: A vector with the same length as `times` variable showing predictive area under the curve at each given time point using Bayesian Model averaging. ## Examples ```r ### Simulating Logistic Regression Data n <- 200 p <- 40 set.seed(123) Sigma <- diag(p) full <- matrix(c(rep(0.5, p*p)), ncol=p) Sigma <- full + 0.5*Sigma cholS <- chol(Sigma) Beta <- c(-1.7,1.8,2.5) X <- matrix(rnorm(n*p), ncol=p) X <- X%*%cholS colnames(X) <- c(paste("gene_",c(1:p),sep="")) beta <- numeric(p) beta[c(1:length(Beta))] <- Beta Xout <- PreProcess(X) X <- Xout$X XB <- X%*%beta probs <- as.vector(exp(XB)/(1+exp(XB))) y <- rbinom(n,1,probs) X <- as.data.frame(X) train_idx <- sample(1:n,0.8*n) test_idx <- setdiff(1:n,train_idx) X_train <- X[train_idx,] y_train <- y[train_idx] bout <- bvs(X_train, y_train, prep=FALSE, family = "logistic", mod_prior = "beta",niter = 50) BMAout <- predBMA(bout, X, y, prep = FALSE, logT = FALSE, train_idx = train_idx, test_idx = test_idx, family="logistic") ### AUC for the prediction: BMAout$auc ### Plotting ROC Curve roc <- BMAout$roc_curve plot(roc,lwd=2,type='l',col='blue') ``` ## References Raftery, A. E., Madigan, D., & Hoeting, J. A. (1997). Bayesian model averaging for linear regression models. Journal of the American Statistical Association, 92(437), 179-191. Nikooienejad, A., Wang, W., & Johnson, V. E. (2020). Bayesian variable selection for survival data using inverse moment priors. Annals of Applied Statistics, 14(2), 809-828. Uno, H., Cai, T., Tian, L., & Wei, L. J. (2007). Evaluating prediction rules for t-year survivors with censored regression models. Journal of the American Statistical Association, 102(478), 527-537. ## Author(s) Amir Nikooienejad

predBMA function