# Classification Accuracy and Consistency Using Lee's (2010) Approach This function computes classification accuracy and consistency indices for complex assessments based on the method proposed by Lee (2010). This function supports both dichotomous and polytomous item response theory (IRT) models. ```r cac_lee(x, cutscore, theta = NULL, weights = NULL, D = 1, cut.obs = TRUE) ``` ## Arguments - `x`: A data frame containing item metadata (e.g., item parameters, number of categories, IRT model types, etc.). See `est_irt()` or `simdat()` for more details about the item metadata. This data frame can be easily created using the `shape_df()` function. - `cutscore`: A numeric vector specifying the cut scores for classification. Cut scores are the points that separate different performance categories (e.g., pass vs. fail, or different grades). - `theta`: A numeric vector of ability estimates. Ability estimates (theta values) are the individual proficiency estimates obtained from the IRT model. The theta parameter is optional and can be `NULL`. - `weights`: An optional two-column data frame or matrix where the first column is the quadrature points (nodes) and the second column is the corresponding weights. This is typically used in quadrature-based IRT analysis. - `D`: A scaling constant used in IRT models to make the logistic function closely approximate the normal ogive function. A value of 1.7 is commonly used for this purpose. Default is 1. - `cut.obs`: Logical. If `TRUE`, it indicates the cutscores on the observed-summed score metric. If `FALSE`, it indicates they are on the IRT theta score metric. Default is `TRUE`. ## Returns A list containing the following elements: * confusion: A confusion matrix showing the cross table between true and expected levels. * marginal: A data frame showing the marginal classification accuracy and consistency indices. * conditional: A data frame showing the conditional classification accuracy and consistency indices. * prob.level: A data frame showing the probability of being assigned to each level category. * cutscore: A numeric vector showing the cut scores used in the analysis. ## Details This function first validates the input arguments. If both `theta` and `weights` are `NULL`, the function will stop and return an error message. Either `theta` (a vector of ability estimates) or `weights` (a quadrature-based weight matrix) must be specified. If `cut.obs = FALSE`, the provided cut scores are assumed to be on the theta (ability) scale, and they are internally converted to the observed summed score scale using the test characteristic curve (TCC). This transformation allows classification to be carried out on the summed score metric, even if theta-based cut points are provided. When `weights` are provided (D method), the function uses the Lord-Wingersky recursive algorithm to compute the conditional distribution of observed total scores at each node. These conditional distributions are used to compute: * the probability of being classified into each performance level, * conditional classification accuracy (probability of correct classification), and * conditional classification consistency (probability of being assigned to the same level upon repeated testing). When `theta` values are provided instead (P method), the same logic applies, but using an empirical distribution of examinees instead of quadrature-based integration. In this case, uniform weights are assigned to all examinees. Finally, marginal classification accuracy and consistency are computed as weighted averages of the conditional statistics across the ability distribution. ## Examples ```r ## -------------------------------------------------------------------------- ## 1. When the empirical ability distribution of the population is available ## (D method) ## -------------------------------------------------------------------------- # Import the "-prm.txt" output file from flexMIRT flex_prm <- system.file("extdata", "flexmirt_sample-prm.txt", package = "irtQ") # Read item parameter data and convert it to item metadata x <- bring.flexmirt(file = flex_prm, "par")$Group1$full_df # Set the cut scores on the observed summed score scale cutscore <- c(10, 20, 30, 50) # Create a data frame containing the quadrature points and corresponding weights node <- seq(-4, 4, 0.25) weights <- gen.weight(dist = "norm", mu = 0, sigma = 1, theta = node) # Calculate classification accuracy and consistency cac_1 <- cac_lee(x = x, cutscore = cutscore, weights = weights, D = 1) print(cac_1) ## ------------------------------------------------------------- ## 2. When individual ability estimates are available (P method) ## ------------------------------------------------------------- # Randomly draw true ability values from N(0, 1) set.seed(12) theta <- rnorm(n = 1000, mean = 0, sd = 1) # Simulate item response data data <- simdat(x = x, theta = theta, D = 1) # Estimate ability parameters using maximum likelihood (ML) est_th <- est_score( x = x, data = data, D = 1, method = "ML", range = c(-4, 4), se = FALSE )$est.theta # Calculate classification accuracy and consistency cac_2 <- cac_lee(x = x, cutscore = cutscore, theta = est_th, D = 1) print(cac_2) ## --------------------------------------------------------- ## 3. When individual ability estimates are available, ## but cut scores are specified on the IRT theta scale ## --------------------------------------------------------- # Set the cut scores on the theta scale cutscore <- c(-2, -0.4, 0.2, 1.0) # Calculate classification accuracy and consistency cac_3 <- cac_lee( x = x, cutscore = cutscore, theta = est_th, D = 1, cut.obs = FALSE ) print(cac_3) ``` ## References Lee, W. C. (2010). Classification consistency and accuracy for complex assessments using item response theory. **Journal of Educational Measurement, 47**(1), 1-17. ## See Also `gen.weight()`, `est_score()`, `cac_rud()` ## Author(s) Hwanggyu Lim hglim83@gmail.com

cac_lee function