atalp function

Automated Test Assembly via Linear Constrained Programming

Ingests item metadata jointly with target test form constraints, and can be parametarized to uses either Boolean (0-1) Integer Linear Programming (ILP) or Mixed Integer Linear Programming (MILP) to construct a test form based on the desired objectives. When MILP is desired the selection of the objective function type should be changed.

atalp( ipool,
      id,
      constraints,
      refine = FALSE,
      permutate = FALSE,
      sorttimes = 999,
      tieselect = -1,
      type = "const",
      verbose = TRUE,
      aprioriadd = NA,
      posthocadd = NA )

Arguments

• ipool: Item by characteristic (property) metadata pool.
• id: Name of unique item identifier.
• constraints: Complex list object identifying the constraints to be applied in the ATA (see makeconstobj for guided process).
• refine: Creates a final test form from permutated solutions, refined to attempt a deviation balance between the observed form and the constraints. Option only effective if permutate is TRUE and type = const in which the constraint weights have meaning; Default is FALSE.
• permutate: Requests the test form to be assembled by resorting (sorttimes) the metadata and selecting the most frequently occurring item combination satisfying the constraints. Relevant only for type = const; Default is FALSE.
• sorttimes: Number of how often the original input metadata should be resorted. Only functional if permutate is TRUE and type = const; default 999, so that sorttimes + main analysis account for a total of 1,000 selection versions.
• tieselect: How should tied items be resolved: -1 (default) - do not manipulate items (which allows for identically functioning items to be included), 1 - select the first item in the list of candidates (sensitive to data sorting); not applicable for situations with all categorical constraints only, 0 - randomly select candidate; not applicable for situations with all categorical constraints only.
• type: Type of objective function: const - constraint based only (default), parmin - constraint + minimum non-categorical parameter combination, parmax - constraint + maximum non-categorical parameter combination.
• verbose: Should progress be printed to the console? Default TRUE.
• aprioriadd: Force item addition (via IDs) to test form before ATA, which affects item selection and constraint attainment success (currently not available).
• posthocadd: Force item addition (via IDs) to test form after ATA, which affects final form specifications (currently not available).

Returns

A complex list object with test assembly specific estimates: - objective: Constrained objective function value.

• items_removed: Removed items from item pool when tieselect is not -1.

• excluded: Items from pool excluded.

• excluded_set: Item sets excluded. Only included if input constobj includes a set_id.

• included: Items from pool included in new test form.

• included_set: Item sets from pool included in new test form. Only included if input constobj includes a set_id.

• final_ids: Final item ids in the test form.

• final_setids: Final set ids in the test form. Only included if input constobj includes a set_id.

Examples

# Specifying constraints
constin <- list(
  nI = 5,                                     # Number of items on the future test
  nC = 4,                                     # Number of constraints to be satisfied
  nameC = c("Content_A","Content_B","p","iSx"), # Name of constraint; must be numeric and must
  # reflect variable name in input
  lowerC = c(2, 3, 3.00, 0.50),               # Lower bound total constraint value on ATA form
  upperC = c(2, 3, 3.50, 0.60),               # Upper bound total constraint value on ATA form
  wC = c(1, 1, 1, 1),                         # Constraint weight used for weighted sum of
                                              # (positive) deviations St
  set_id = NA                                 # Aggregation ID for units / sets 
)

# Running atalp
testLP <- atalp(ipool = metadata_example,
                id = "Item",
                constraints = constin)

# Summary of results
summary(testLP)

References

Chen, P. (2017). Should we stop developing heuristics and only rely on mixed integer programming solvers in automated test assembly? Applied Psychological Measurement, 41, 227-240.

Diao, Q., & van der Linden, W. J. (2011). Automated test assembly using lp_Solve Version 5.5 in R. Applied Psychological Measurement, 35, 398-409.

Shao, C., Liu, S., Yang, H., & Tsai, T. (2019). Automated test assembly using SAS operations research software in a medical licensing examination. Applied Psychological Measurement, 00, 1-15.

van der Linden, W. J. (2005). A comparison of item-selection methods for adaptive tests with content constraints. Journal of Educational Measurement, 42, 283-302.

Author(s)

Michael Chajewski (mchajewski@hotmail.com), Gulsah Gurkan (gurkangulsah@gmail.com)