# Random generation of multistage tests (dichotomous and polytomous models) This command generates a response pattern to a multistage test, for a given item bank (with either dichotomous or polytomous models), an MST structure for modules and stages, a true ability level, and several lists of MST parameters. ```r randomMST(trueTheta, itemBank, modules, transMatrix, model = NULL, responses = NULL, genSeed = NULL, start = list(fixModule = NULL, seed = NULL, theta = 0, D = 1), test = list(method = "BM", priorDist = "norm", priorPar = c(0, 1), range = c(-4, 4), D = 1, parInt = c(-4, 4, 33), moduleSelect = "MFI", constantPatt = NULL, cutoff = NULL, randomesque = 1, random.seed = NULL, score.range = "all"), final = list(method = "BM", priorDist = "norm", priorPar = c(0, 1), range = c(-4, 4), D = 1, parInt = c(-4, 4, 33), alpha = 0.05), allTheta = FALSE, save.output = FALSE, output = c("path", "name", "csv")) ## S3 method for class 'mst' print(x, ...) ## S3 method for class 'mst' plot(x, show.path = TRUE, border.col = "red", arrow.col = "red", module.names = NULL, save.plot = FALSE, save.options = c("path", "name", "pdf"),...) ``` ## Arguments - `trueTheta`: numeric: the value of the true ability level. - `itemBank`: numeric: a suitable matrix of item parameters. See Details . - `modules`: a binary matrix that specifies the item membership to the modules. See Details . - `transMatrix`: a binary squared matrix representing the structure of the MST and the transitions between the moduels and the stages. See Details . - `model`: either `NULL` (default) for dichotomous models, or any suitable acronym for polytomous models. Possible values are `"GRM"`, `"MGRM"`, `"PCM"`, `"GPCM"`, `"RSM"` and `"NRM"`. See Details . - `responses`: either `NULL` (default) or a vector of pre-specified item responses with as many components as the rows of `itemBank`. See Details . - `genSeed`: either a numeric value to fix the random generation of responses pattern or `NULL` (default). Ignored if `responses` is not `NULL`. See Details . - `start`: a list with the options for starting the multistage test. See Details . - `test`: a list with the options for provisional ability estimation and next module selection. See Details . - `final`: a list with the options for final ability estimation or scoring. See Details . - `allTheta`: logical: should all provisional ability estimates and standard errors be computed and returned (even within each module)? Default is `FALSE`, meaning that provisional ability estimates and standard errors are computed only at the end of each module administration. - `save.output`: logical: should the output be saved in an external text file? (default is `FALSE`). - `output`: character: a vector of three components. The first component is either the file path to save the output of `"path"` (default), the second component is the name of the output file, and the third component is the file type, either `"txt"` or `"csv"` (default). See Details . - `x`: either an object of class "mst", typically an output of `randomMST` function, or a transition matrix (for `plot.mst()` function only). - `show.path`: logical: should the selected path (i.e. set of successive modules) be highlighted in the plot (default is TRUE)? - `border.col`: character: the color for the rectangle border of the path (i.e. selected modules). Default is `"red"`. Ignored if `show.path` is `FALSE`. - `arrow.col`: character: the color for the connecting arrows in the path (i.e. between selected modules). Default is `"red"`. Ignored if `show.path` is `FALSE`. - `module.names`: either `NULL` (default) or a vector of character names for the modules. See Details . - `save.plot`: logical: should the plot be saved in an external figure? (default is `FALSE`). - `save.options`: character: a vector of three components. The first component is either the file path or `"path"` (default), the second component is the name of the output file or ,`"name"` (default), and the third component is the file extension, either `"pdf"` (default) or `"jpeg"`. Ignored if `save.plot` is `FALSE`. See Details . - `...`: other generic arguments to be passed to `print` and `plot` functions. ## Returns The function `randomMST` returns a list of class "mst" with the following arguments: - **trueTheta**: the value of the `trueTheta` argument. - **selected.modules**: a vector with the modules (identified by their position in the transition matrix) that were selected for the MST. - **items.per.module**: a vector with the number of items per selected module (in the same order as in `selected.modules`). - **transMatrix**: the value of the `transMatrix` argument. - **model**: the value of the `model` argument. - **testItems**: a vector with the items that were administered during the test. - **itemPar**: a matrix with the parameters of the items administered during the test. - **pattern**: the generated (or selected) response pattern (as vector of 0 and 1 entries for dichotomous items or positive integer values for polytomous items). - **thetaProv**: a vector with the provisional ability estimates (or test scores if `test$method` is `"score"`). - **seProv**: a vector with the standard errors of the provisional ability estimates (or vector of `NA`'s if `test$method` is `"score"`). - **thFinal**: the final ability estimate (or test score if `test$method` is `"score"`). - **seFinal**: the standard error of the final ability estimate (or `NA` if `test$method` is `"score"`). - **ciFinal**: the confidence interval of the final ability estimate (or `c(NA, NA)` if `test$method` is `"score"`). - **genSeed**: the value of the `genSeed` argument. - **startFixModule**: the value of the `start$fixModule` argument (or its default value if missing). - **startSeed**: the value of the `start$seed` argument (or its default value if missing). - **startTheta**: the value of the `start$theta` argument (or its default value if missing). - **startD**: the value of the `start$D` argument (or its default value if missing). - **startThStart**: the starting ability value used for selecting the first module of the test. - **startSelect**: the value of the `start$startSelect` argument (or its default value if missing). - **provMethod**: the value of the `test$method` argument (or its default value if missing). - **provDist**: the value of the `test$priorDist` argument (or its default value if missing). - **provPar**: the value of the `test$priorPar` argument (or its default value if missing). - **provRange**: the value of the `test$range` argument (or its default value if missing). - **provD**: the value of the `test$D` argument (or its default value if missing)or `NA` if `model` is not `NULL`. - **moduleSelect**: the value of the `test$moduleSelect` argument (or its default value if missing). - **constantPattern**: the value of the `test$constantPatt` argument (or its default value if missing). - **cutoff**: the value of the `test$cutoff` argument (or its default value if missing). - **randomesque**: the value of the `test$randomesque` argument (or its default value if missing). - **random.seed**: the value of the `test$random.seed` argument (or its default value if missing). - **score.range**: the value of the `test$score.range` argument (or its default value if missing). - **best.module**: a vector of boolean values indicating whether the optimal modules were selected or not. - **finalMethod**: the value of the `final$method` argument (or its default value if missing). - **finalDist**: the value of the `final$priorDist` argument (or its default value if missing). - **finalPar**: the value of the `final$priorPar` argument (or its default value if missing). - **finalRange**: the value of the `final$range` argument (or its default value if missing). - **finalD**: the value of the `final$D` argument (or its default value if missing), or `NA` if `model` is not `NULL`. - **finalAlpha**: the value of the `final$alpha` argument (or its default value if missing). - **save.output**: the value of the `save.output` argument. - **output**: the value of the `output` argument. - **assigned.responses**: a logical value, being `TRUE` if `responses` was provided or `FALSE` `responses` was set to `NULL`. - **allTheta**: either a table with all ad-interim ability estimates (even within module, in the CAT spirit) if `allTheta` is set to `TRUE`, or `NULL` if `allTheta` is set to `FALSE`. - **assigned.responses**: the value of the `responses` argument (or its default value if missing). The function `print.mst` returns similar (but differently organized) results. ## Details The `randomMST` function generates a multistage test using an item bank specified by arguments `itemBank` and `model`, an MST structure provided by arguments `modules` and `transMatrix`, and for a given true ability level specified by argument `trueTheta`. Dichotomous IRT models are considered whenever `model` is set to `NULL` (default value). In this case, `itemBank` must be a matrix with one row per item and four columns, with the values of the discrimination, the difficulty, the pseudo-guessing and the inattention parameters (in this order). These are the parameters of the four-parameter logistic (4PL) model (Barton and Lord, 1981). See `genDichoMatrix` for further information. Polytomous IRT models are specified by their respective acronym: `"GRM"` for Graded Response Model, `"MGRM"` for Modified Graded Response Model, `"PCM"` for Partical Credit Model, `"GPCM"` for Generalized Partial Credit Model, `"RSM"` for Rating Scale Model and `"NRM"` for Nominal Response Model. The `itemBank` still holds one row per item, end the number of columns and their content depends on the model. See `genPolyMatrix` for further information and illustrative examples of suitable polytomous item banks. The `modules` argument must be a binary 0/1 matrix with as many rows as the item bank `itemBank` and as many columns as the number of modules. Values of 1 indicate to which module(s) the items belong to, i.e. a value of 1 on row **i** and column **j** means that the **i**-th item belongs to the **j**-th module. The `transMatrix` argument must be a binary 0/1 square matrix with as many rows (and columns) as the number of modules. All values of 1 indicate the possible transitions from one module to another, i.e. a value of 1 on row **i** and column **j** means that the MST can move from **i**-th module to **j**-th module. By default all item responses will be randomly drawn from parent distribution set by the item bank parameters of the `itemBank` matrix (using the `genPattern` function for instance). Moreover, the random generation of the item responses can be fixed (for e.g., replication purposes) by assigning some numeric value to the `genSeed` argument. By default this argument is equal to `NULL` so the random seed is not fixed (and two successive runs of `randomMST` will usually lead to different response patterns). It is possible, however, to provide a full response pattern of previously recorded responses to each item of the item bank, for instance for post-hoc simulations. This is done by providing to the `responses` argument a vector of binary entries (without missing values). By default `responses` is set to `NULL` and item responses will be drawn from the item bank parameters. The test specification is made by means of three lists of options: one list for the selection of the starting module, one list with the options for provisional ability estimation and next module selection, and one list with the options for final ability estimation. These lists are specified respectively by the arguments `start`, `test` and `final`. The `start` list can contain one or several of the following arguments: * `fixModule`: either an integer value, setting the module to be administered as first stage (as its row number in te `transMatrix` argument for instance), or `NULL` (default) to let the function select the module. * `seed`: either a numeric value to fix the random seed for module selection, `NA` to randomly select the module withour fixing the random seed, or `NULL` (default) to make random module selection without fixing the random seed. Ignored if `fixModule` is not `NULL`. * `theta`: the initial ability value, used to select the most informative module at this ability level (default is 0). Ignored if either `fixModule` or `seed` is not `NULL`. See `startModule` for further details. * `D`: numeric, the metric constant. Default is `D=1` (for logistic metric); `D=1.702` yields approximately the normal metric (Haley, 1952). Ignored if `model` is not `NULL` and if `startSelect` is not `"MFI"`. These arguments are passed to the function `startModule` to select the first module of the multistage test. The `test` list can contain one or several of the following arguments: * `method`: a character string to specify the method for ability estimation or scoring. Possible values are: `"BM"` (default) for Bayesian modal estimation (Birnbaum, 1969), `"ML"` for maximum likelihood estimation (Lord, 1980), `"EAP"` for expected a posteriori (EAP) estimation (Bock and Mislevy, 1982), and `"WL"` for weighted likelihood estimation (Warm, 1989). The `method` argument can also take the value `"score"`, meaning that module selection is based on the test score from the previously administered modules. The latter works only if `cutoff` argument is supplied appropriately, otherwise this leads to an error message. * `priorDist`: a character string which sets the prior distribution. Possible values are: `"norm"` (default) for normal distribution, `"unif"` for uniform distribution, and `"Jeffreys"` for Jeffreys' noninformative prior distribution (Jeffreys, 1939, 1946). ignored if `method` is neither `"BM"` nor `"EAP"`. * `priorPar`: a vector of two numeric components, which sets the parameters of the prior distribution. If (`method="BM"` or `method=="EAP"`) and `priorDist="norm"`, the components of `priorPar` are respectively the mean and the standard deviation of the prior normal density. If (`method="BM"` or `method="EAP"`) and `priorDist="unif"`, the components of `priorPar` are respectively the lower and upper bound of the prior uniform density. Ignored in all other cases. By default, `priorPar` takes the parameters of the prior standard normal distribution (i.e., `priorPar=c(0,1)`). In addition, `priorPar` also provides the prior parameters for the comoutation of MLWI and MPWI values for next item selection (see `nextModule` for further details). * `range`: the maximal range of ability levels, set as a vector of two numeric components. The ability estimate will always lie to this interval (set by default to [-4, 4]). Ignored if `method=="EAP"`. * `D`: the value of the metric constant. Default is `D=1` for logistic metric. Setting `D=1.702` yields approximately the normal metric (Haley, 1952). Ignored if `model` is not `NULL`. * `parInt`: a numeric vector of three components, holding respectively the values of the arguments `lower`, `upper` and `nqp` of the `eapEst`, `eapSem` and `MWI` commands. It specifies the range of quadrature points for numerical integration, and is used for computing the EAP estimate, its standard error, and the MLWI and MPWI values for next item selection. Default vector is (-4, 4, 33), thus setting the range from -4 to 4 by steps of 0.25. Ignored if `method` is not `"EAP"` and if `itemSelect` is neither `"MLMWI"` nor `"MPMWI"`. * `moduleSelect`: the rule for next module selecion, with possible values: 1. `"MFI"` (default) for maximum Fisher information criterion; 2. `"MLMWI"` for maximum likelihood weighted (module) information criterion; 3. `"MPMWI"` for posterior weighted (module) information criterion; 4. `"MKL"` for (module) Kullback-Leibler information methods; 5. `"MKLP"` for posterior (module) Kullback-Leibler information methods; 6. `"random"` for random selection. This argument is ignored if `cutoff` is supplied appropriately. See `nextModule` for further details. * `constantPatt`: the method to estimate ability in case of constant pattern (i.e. only correct or only incorrect responses). Can be either `NULL` (default), `"BM"`, `"EAP"`, `"WL"`, `"fixed4"` (for fixed stepsize adjustment with step 0.4), `"fixed7"` (for fixed stepsize adjustment with step 0.7), or `"var"` (for variable stepsize adjustment). This is currently implemented only for dichotomous IRT models and is sgnored if `method` is `"score"`. See `thetaEst` for further details. * `cutoff`: either `NULL` (default) or a suitable matrix of thresholds to select the next module. Thresholds can reflect module selection based on ability estimation (and then `method` should hold one of the ability estimation methods) or on provisional test score (and then `method` must be set to `"score"`. See `nextModule` for further details about suitable definition of the `cutoff` matrix (and the examples below). * `randomesque`: a probability value to select the optimal module. Default is one, so the optimal module is always chosen. With a value smaler than one, other elligible modules can be selected. * `random.seed`: either `NULL` (default) or a numeric value to fix the random seed of randomesque selection of the module. Ignored if `randomesque` is equal to one. * `score.range`: a character value that specifies on which set of modules the provisional test score should be computed. Possible values are `"all"` (default) to compute the score with all previously administered modules, or `"last"` to compute the score only with the last module. Ignored if `method` is not `"score"`. These arguments are passed to the functions `thetaEst` and `semTheta` to estimate the ability level and the standard error of this estimate. In addition, some arguments are passed to `nextModule` to select the next module appropriately. Finally, the `final` list can contain the arguments `method`, `priorDist`, `priorPar`, `range`, `D` and `parInt` of the `test` list (with possiblly different values), as well as the additional `alpha` argument. The latter specifies the $\alpha$ level of the final confidence interval of ability, which is computed as $$ [\hat{\theta}-z_{1-\alpha/2} \; se(\hat{\theta}) ; \hat{\theta}+z_{1-\alpha/2} \; se(\hat{\theta})] $$ where $\hat{\theta}$ and $se(\hat{\theta})$ are respectively the ability estimate and its standard error. If some arguments of these lists are missing, they are automatically set to their default value. Usually the ability estimates and related standard errors are computed right after the full administration of each module (that is, if current module has **k** items, the **(k-1)** ability levels and standard errors from the first administered **(k-1)** are not computed). This can however be avoided by fixing the argument `allTheta` to `TRUE` (by default it is `FALSE`). In this case, all provisional ability estimates (or test scores) and standard errors (or `NA`'s) are computed and returned. The output of `randomMST`, as displayed by the `print.mst` function, can be stored in a text file provided that `save.output` is set to `TRUE` (the default value `FALSE` does not execute the storage). In this case, the `(output` argument mus hold three character values: the path to where the output file must be stored, the name of the output file, and the type of output file. If the path is not provided (i.e. left to its default value `"path"`), it will be saved in the default working directory. The default name is `"name"`, and the default type is `"csv"`. Any other value yields a text file. See the Examples section for an illustration. The function `plot.mst` represents the whole MST structure with as many rectangles as there are available modules, arrows connecting all the modules according to the `transMatrix` structure. Each stage is displayed as one horizontal layout with stage 1 on the top and final stage at the bottom of the figure. The selected path (i.e. set of modules) is displayed on the plot when `show.path` is `TRUE` (which is the default value). Modules from the path and arrows between them are then highlighted in red (by default), and these colors can be modified by settong `border.col` and `arrow.col` arguments with appropriate color names. By default, modules are labelled as ``module 1", ``module 2" etc., the numbering starting from left module to right module and from stage 1 to last stage. These labels can be modified by providing a vector of character names to argument `module.names`. This vector must have as many components as the total number of modules and being ordered identically as described above. Note that the MST structure can be graphically displayed by only providing (as `x` argument) the transition matrix of the MST. In this case, `show.path` argument is ignored. This is useful to represent the MST structure set by the transition matrix without running an MST simulation. Finally, the plot can be saved in an external file, either as PDF or JPEG format. First, the argument `save.plot`must be set to `TRUE` (default is `FALSE`). Then, the file path for figure storage, the name of the figure and its format are specified through the argument `save.options`, all as character strings. See the `Examples` section for further information and a practical example. ## References Barton, M.A., and Lord, F.M. (1981). **An upper asymptote for the three-parameter logistic item-response model**. Research Bulletin 81-20. Princeton, NJ: Educational Testing Service. Birnbaum, A. (1969). Statistical theory for logistic mental test models with a prior distribution of ability. **Journal of Mathematical Psychology, 6**, 258-276. doi: 10.1016/0022-2496(69)90005-4 Bock, R. D., and Mislevy, R. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. **Applied Psychological Measurement, 6**, 431-444. doi: 10.1177/014662168200600405 Haley, D.C. (1952). **Estimation of the dosage mortality relationship when the dose is subject to error**. Technical report no 15. Palo Alto, CA: Applied Mathematics and Statistics Laboratory, Stanford University. Jeffreys, H. (1939). **Theory of probability**. Oxford, UK: Oxford University Press. Jeffreys, H. (1946). An invariant form for the prior probability in estimation problems. **Proceedings of the Royal Society of London. Series A, Mathematical and Physical Sciences, 186**, 453-461. Lord, F. M. (1980). **Applications of item response theory to practical testing problems**. Hillsdale, NJ: Lawrence Erlbaum. Warm, T.A. (1989). Weighted likelihood estimation of ability in item response models. **Psychometrika, 54**, 427-450. doi: 10.1007/BF02294627 ## Author(s) David Magis Department of Psychology, University of Liege, Belgium david.magis@uliege.be Duanli Yan Educational Testing Service, Princeton, USA dyan@ets.org ## See Also `thetaEst`, `semTheta`, `eapEst`, `eapSem`, `genPattern`, `genDichoMatrix` , `genPolyMatrix` , `nextModule` ## Examples ```r ## Dichotomous models ## # Generation of an item bank under 2PL, made of 7 successive modules that target # different average ability levels and with different lengths # (the first generated item parameters hold two modules of 8 items each) it <- rbind(genDichoMatrix(16, model = "2PL"), genDichoMatrix(6, model = "2PL", bPrior = c("norm", -1, 1)), genDichoMatrix(6, model = "2PL", bPrior = c("norm", 1, 1)), genDichoMatrix(9, model = "2PL", bPrior = c("norm", -2, 1)), genDichoMatrix(9, model = "2PL", bPrior = c("norm", 0, 1)), genDichoMatrix(9, model = "2PL", bPrior = c("norm", 2, 1))) it <- as.matrix(it) # Creation of the 'modules' matrix to list item membership in each module modules <- matrix(0, 55, 7) modules[1:8, 1] <- modules[9:16, 2] <- modules[17:22, 3] <- 1 modules[23:28, 4] <- modules[29:37, 5] <- modules[38:46, 6] <- 1 modules[47:55, 7] <- 1 # Creation of the transition matrix to define a 1-2-3 MST trans <- matrix(0, 7, 7) trans[1, 3:4] <- trans[2, 3:4] <- trans[3, 5:7] <- trans[4, 5:7] <- 1 # Creation of the start list: selection by MFI with ability level 0 start <- list(theta = 0) # Creation of the test list: module selection by MFI, ability estimation by WL, # stepsize .4 adjustment for constant pattern test <- list(method = "WL", moduleSelect = "MFI", constantPatt = "fixed4") # Creation of the final list: ability estimation by ML final <- list(method = "ML") # Random MST generation for true ability level 1 and all ad-interim ability estimates res <- randomMST(trueTheta = 1, itemBank = it, modules = modules, transMatrix = trans, start = start, test = test, final = final, allTheta = TRUE) # Module selection by cut-scores for ability estimates # Creation of cut-off scores for ability levels: cut score 0 between modules 3 and 4 # and cut scores -1 and 1 between modules 5, 6 and 7 # randomesque selection with probability .8 cut <- rbind(c(3, 4, 0), c(5, 6, -1), c(6, 7, 1)) test <- list(method = "WL", constantPatt = "fixed4", cutoff = cut, randomesque = 0.8) res <- randomMST(trueTheta = 1, itemBank = it, modules = modules, transMatrix = trans, start = start, test = test, final = final, allTheta = TRUE) # Module selection by cut-scores for test scores # Creation of cut-off scores for test scores: cut score 4 between modules 3 and 4 # and cut scores 5 and 9 between modules 5, 6 and 7 cut.score <- rbind(c(3, 4, 4), c(5, 6, 5), c(6, 7, 9)) test <- list(method = "score", cutoff = cut.score) final <- list(method = "score") res <- randomMST(trueTheta = 1, itemBank = it, modules = modules, transMatrix = trans, start = start, test = test, final = final, allTheta = TRUE) # Modification of cut-scores of stage 3 to use only the last module from stage 2 (6 items): # cut scores 2 and 4 between modules 5, 6 and 7 cut.score2 <- rbind(c(3, 4, 4), c(5, 6, 2), c(6, 7, 4)) test <- list(method = "score", cutoff = cut.score2, score.range = "last") final <- list(method = "score") res <- randomMST(trueTheta = 1, itemBank = it, modules = modules, transMatrix = trans, start = start, test = test, final = final, allTheta = TRUE) ## Plot options plot(trans) plot(res) plot(res, show.path = FALSE) plot(res, border.col = "blue") plot(res, arrow.col = "green") ```

randomMST function