# Simulation of multiple examinees of adaptive tests This command runs a set of adaptive tests, for a given item bank, a set of ability levels, a possible matrix of item responses, and several lists of CAT parameters (starting items, stopping rule, provisional and final ability estimators). ```r simulateRespondents(thetas, itemBank, responsesMatrix = NULL, model = NULL, genSeed = NULL, cbControl = NULL, rmax = 1, Mrmax = "restricted", start = list(fixItems = NULL, seed = NULL, nrItems = 1, theta = 0, D = 1, randomesque = 1, random.seed = NULL, startSelect = "MFI", cb.control = FALSE, random.cb = NULL), test = list(method = "BM", priorDist = "norm", priorPar = c(0,1), weight = "Huber", tuCo = 1, sem.type = "classic", sem.exact = FALSE, se.ase = 10, range = c(-4, 4), D = 1, parInt = c(-4, 4, 33), itemSelect = "MFI", infoType = "observed", randomesque = 1, random.seed = NULL, AP = 1, proRule = "length", proThr = 20, constantPatt = NULL), stop = list(rule = "length", thr = 20, alpha = 0.05), final = list(method = "BM", priorDist = "norm", priorPar = c(0,1), weight = "Huber", tuCo = 1, sem.type = "classic", sem.exact = FALSE, range = c(-4, 4), D = 1, parInt = c(-4, 4, 33), alpha = 0.05), save.output = FALSE, output = c("", "catR", "csv")) ## S3 method for class 'catResult' print(x, ...) ## S3 method for class 'catResult' plot(x, type = "all", deciles = "theta", save.plot = FALSE, save.options = c("", "plot", "pdf"), res = 300, ...) ``` ## Arguments - `thetas`: numeric: a vector of true ability values for which a CAT must be generated for each component. - `itemBank`: numeric: a suitable matrix of item parameters (possibly augmented by group membership for content balancing). See Details . - `responsesMatrix`: numeric: either `NULL` (default) or a suitable matrix of item responses. See Details . - `model`: either `NULL` (default) for dichotomous models, or any suitable acronym for polytomous models. Possible values are `"GRM"`, `"MGRM"`, `"PCM"`, `"GPCM"`, `"RSM"` and `"NRM"`. See Details . - `genSeed`: either a vector of numeric values to fix the random seed of each generated pattern, or `NULL` (default). Ignored if `responsesMatrix` is not `NULL`. See Details . - `cbControl`: either a list of accurate format to control for content balancing, or `NULL`. See Details . - `rmax`: numeric: the maximum exposure rate (default is 1). - `Mrmax`: character: the method for controlling maximum exposure rate. Possible values are `"restricted"`(default) or `"IE"`. See Details . - `start`: a list with the options for starting the adaptive test. See Details . - `test`: a list with the options for provisional ability estimation and next item selection. See Details . - `stop`: a list with the options of the stopping rule. See Details . - `final`: a list with the options for final ability estimation. See Details . - `save.output`: logical: should the output be saved in an external text file? (default is `FALSE`). - `output`: character: a vector of three components. The first component is either the file path to save the output or `""` (default), the second component is either the initial part of the name of the output file or `"catR"`, and the third component is the file type, either `"txt"` or `"csv"` (default). See Details . - `x`: an object of class "cat", typically an output of `simulateRespondents` function. - `type`: character: the type of plot to display. Possible values are `"all"` (default), `"trueEst"`, `"expRate"`, `"cumExpRate"`, `"cumNumberItems"`, `"expRatePara"`, `"condBias"`, `"condRMSE"`, `"numberItems"`, `"sError"` and `="condThr"`. See Details . - `deciles`: whether the deciles number (`"deciles"`) or the mean ability level per decile (`"theta"`) will be used in the axis of the plots. - `save.plot`: logical: should the plot be saved in an external figure? (default is `FALSE`). - `save.options`: character: a vector of three components. The first component is either the file path or `""` (default), the second component is the name of the output file or ,`"plot"` (default), and the third component is the file extension, either `"pdf"` (default) or `"jpeg"`. Ignored if `save.plot` is `FALSE`. See Details . - `res`: numeric: the resolution for JPEG figures (default value is 300). - `...`: other generic arguments to be passed to `print` and `plot` functions. ## Returns The function `simulateRespondents` returns a list of class "catResult" with the following arguments: - **thetas**: the value of the `thetas` argument. - **itemBank**: the value of the `itemBank` argument. - **responsesMatrix**: the value of the `responsesMatrix` argument. - **model**: the value of the `model` argument. - **genSeed**: the value of the `genSeed` argument. - **cbControl**: the value of the `cbControl` argument. - **rmax**: the value of the `rmax` argument. - **Mrmax**: the value of the `Mrmax` argument. - **start**: the value of the `start` argument. - **test**: the value of the `test` argument. - **stop**: the value of the `stop` argument. - **final**: the value of the `final` argument. - **save.output**: the value of the `save.output` argument. - **output**: the value of the `output` argument. - **estimatedThetas**: a vector with (final) estimated ability levels. - **correlation**: the correlation between the `thetas` vector and estimated ability levels. - **bias**: the value of the bias between true and estimated ability levels. - **RMSE**: the value of the RMSE between true and estimated ability levels. - **thrOK**: a vector indicating whether the respondents finished the test satisfying (1) or not (0) the stop criteria. - **exposureRates**: a vector with empirical exposure rates of all item in the bank. - **testLength**: the mean test length. - **overlap**: the item overlap rate. - **numberItems**: a vector with the lengths of each adaptive test (i.e. the number of items administered). - **condTheta**: a vector with the mean ability level per decile. - **condBias**: a vector with conditional mean bias per decile. - **condRMSE**: a vector with conditional RMSE per decile. - **condnItems**: a vector with conditional mean test length per decile. - **condSE**: a vector with conditional mean standard error per decile. - **condthrOK**: a vector with conditional proportion of respondents that finish the test satisfying the stop criteria per decile. - **ndecile**: a vector with the number of respondents per decile. - **final.values.df**: a data frame with true ability levels, final ability estimates and standard errors, and test lengths. - **responses.df**: a data frame with all items administered, all item responses and all provisional ability estimates. -99 is displayed when the actual test length is smaller than the maximum test length for those item positions where no item was administered as the stop criterion was already reached. - **start.time**: the CPU time at the start of the CAT generation. - **finish.time**: the CPU time at the end of all CAT generations. ## Details The `simulateRespondents` function permits to generate several adaptive tests to a set of respondents defined by their ability levels. It makes a repeated call to an adaptive test using an item bank specified by arguments `itemBank` `model`, and with the same `start`, `test`, `stop` and `final` lists. Content balancing can also be controlled for each respondent with the `cbControl` argument. All arguments of `simulateRespondents` are used in exactly the same manner as in `randomCAT` (so refer to this function for further information), except the following four. First, `thetas` is now a vector of ability levels, and a CAT will be generated for each component of `thetas`. If `responsesMatrix` is `NULL`, item responses are generated from the IRT model, item bank parameters, and the ability levels. In this case, `thetas` can be considered as real ability levels. Otherwise, `responsesMatrix` must be provided as a matrix with as many rows as the length of `thetas` and as many columns as the number of items in `itemBank`. Each row contains the response pattern of one examinee whose ability level is given by the value of the corresponding component of `thetas`. Note that only allowable item responses can be included in `responsesMatrix` and missing values are not accepted. Fixing the random seed can also be done with the `genSeed` argument. The latter must hold as many components as the vector `thetas`, otherwise an error message is returned. Each component of `genSeed` is used to fix the seed for ach pattern generation. The option of providing a response matrix through `responsesMatrix` is considered for two possible uses: 1. **post-hoc simulations**: examinees provided responses to the full item bank and one wants to test the performance of a CAT with those responses, 2. simulations considering **item parameter estimation errors**: responses to the full bank are generated with the real parameters and the CAT is run is the estimated parameters and the responses from the correct model. Note that if `thetas` holds a single value, then the function simply calls `randomCAT` and returns its output instead. Second, `rmax` fixes the desired maximum exposure rate for all items. Default value is 1, allowing thus items to be administered to all respondents without restrictions. Third, the `Mrmax` argument fixes the method to constraint exposure rates to be smaller than the maximum allowed rate. Possible methods are the restricted method (`"restricted"`; Revuelta and Ponsoda, 1998) and the item-eligibility method (`"IE"`; van der Linden and Veldkamp, 2004). A description of both methods can be found in Barrada, Abad and Veldkamp (2009) Fourth, if the length of `thetas` is greater than 1, `save.output` for `randomCAT` is fixed to `"FALSE"`. Otherwise, the same file would be overwritten for each new respondent. The output of `simulateRespondents`, as displayed by the `print.catResult` function, proposes summary statistics related to overall accuracy (bias, RMSE, etc), conditional accuracy per decile, and item exposure control (minimum and maximum exposure rates, test overlap rate, etc) among all generated CATs. This output can be saved when `save.output` is set to `TRUE`, and if so, three outpout files are returned: (1) one with the main summary statistics, with overall results and conditional on decile results; (2) the file with respondents' patterns, items administered and provisional ability estimates; and (3) a table with true and estimated ability levels, final standard errors and numbers of items administered per respondent. Specific information provided and saved depends on the `rule` used in `stop`. This output can also be graphically displayed with the `plot.catResult` function. In addition to the output of the function and the item bank, it takes the argument `type` to determine which plot should be returned. Ten different single plots can be displayed: 1. `"trueEst"`: the scatterplot of true vs. estimated ability levels. 2. `"expRate"`: the exposure rates of the items, having ranked them according to these exposure rates. 3. `"cumExpRate"`: the cumulative exposure rates of the items, having ranked them according to these exposure rates. 4. `"cumNumberItems"`: the test length as a function of cumulative percent of examinees. This plot is not available when `rule` is 'length'. 5. `"expRatePara"`: the scatterplot of item exposure rates vs. item discrimination parameters. This plot is not available when `model` is 'PCM' or 'NRM', as in those IRT models there are no discrimination parameters. 6. `"condBias"`: the conditional bias of ability estimation as a function of the deciles of the true ability levels. 7. `"condRMSE"`: the conditional RMSE of ability estimation as a function of the deciles of the true ability levels. 8. `"numberItems"`: the conditional test length as a function of the deciles of the true ability levels. This plot is not available when `rule` is 'length'. 9. `"sError"`: the conditional standard error of ability estimation as a function of the deciles of the true ability levels. 10. `"condThr"`: the conditional proportions of CATs satisfying the 'precision' or 'classification' stopping rule, as a function of the deciles of the true ability levels. This plot is not available when `rule` is 'length'. In addition, the value `"all"` (default value) displays several available plots in a single panel. Displayed plots depend on the `rule` used in `stop`: (a) with 'length', `"trueEst"`, `"condBias"`, `"condRMSE"`, `"expRate"`, `"cumExpRate"`, `"expRatePara"`; (b) with 'precision' and 'classification', all the plots but `"cumExpRate"`. These plots can be saved as external PDF or JPEG files, by setting `save.plot` to `TRUE` and defining the arguments of `save.options` accurately (see `randomCAT` for further explanations and the Examples section below). ## References Barrada, J. R., Abad, F. J., and Veldkamp, B. P. (2009). Comparison of methods for controlling maximum exposure rates in computerized adaptive testing. **Psicothema**, **21**, 313-320. Magis, D. and Barrada, J. R. (2017). Computerized Adaptive Testing with R: Recent Updates of the Package **catR**. **Journal of Statistical Software**, **Code Snippets**, **76(1)**, 1-18. tools:::Rd_expr_doi("10.18637/jss.v076.c01") Magis, D., and Raiche, G. (2012). Random Generation of Response Patterns under Computerized Adaptive Testing with the R Package **catR**. **Journal of Statistical Software**, **48 (8)**, 1-31. tools:::Rd_expr_doi("10.18637/jss.v048.i08") Revuelta, J., and Ponsoda, V. (1998). A comparison of item exposure control methods in computerized adaptive testing. **Journal of Educational Measurement, 35**, 311-327. tools:::Rd_expr_doi("10.1111/j.1745-3984.1998.tb00541.x") van der Linden, W.J., and Veldkamp, B.P. (2004). Constraining item exposure in computerized adaptive testing with shadow tests. **Journal of Educational and Behavioral Statistics, 29**, 273-291. tools:::Rd_expr_doi("10.3102/10769986029003273") ## Author(s) Juan Ramon Barrada Department of Psychology and Sociology, Universidad Zaragoza, Spain barrada@unizar.es David Magis Department of Psychology, University of Liege, Belgium david.magis@uliege.be Guido Corradi Department of Psychology and Sociology, Universidad Zaragoza, Spain guidocor@gmail.com ## See Also `randomCAT` ## Examples ```r ## Dichotomous IRT model ## # Loading the 'tcals' parameters data(tcals) bank <- as.matrix(tcals[,1:4]) # Creation of a starting list with three theta values start <- list(theta = -1:1, randomesque = 5) # Creation of 'test' list: maximum likelihood estimation and # progressive method test <- list(method = "ML", itemSelect = "progressive") # Creation of a stopping rule: precision criterion, standard # error to be reached 0.3 stop <- list(rule = "precision", thr = 0.3) # Creation of 'final' list: ML estimation of final ability final <- list(method = "ML") # Generation of ten respondents set.seed(1) thetas <- rnorm(10) # Default CAT generations, output not saved res <- simulateRespondents(thetas, bank, start = start, test = test, stop = stop, final = final) # Maximum exposure restricted to 0.8 res2 <- simulateRespondents(thetas, bank, start = start, test = test, stop = stop, final = final, rmax = 0.8) ## Not run: # Output saved res3 <- simulateRespondents(thetas, bank, start = start, test = test, stop = stop, final = final, save.output = TRUE, output = c("C:/Program Files/", "out", "txt")) ## End(Not run) # With content balancing # # Creation of an appropriate list for content balancing # Equal proportions across subgroups of items cbList <- list(names = c("Audio1", "Audio2", "Written1", "Written2", "Written3"), props = c(0.1, 0.2, 0.2, 0.2, 0.3)) # CAT test (same options as above) res4 <- simulateRespondents(thetas, tcals, start = start, test = test, stop = stop, final = final, cbControl = cbList) # Plotting and saving output # # Plotting all possible panels plot(res) plot(res, deciles = "deciles") ## Not run: # Saving the plot in the "fig" pdf file in "c:/Program Files/" plot(res, save.plot = TRUE, save.options = c("c:/Program Files/", "fig", "pdf")) ## End(Not run) # Plotting the 'trueEst' type of plot plot(res, type = "trueEst") ```

simulateRespondents function