Optimal number of strata for the stratified methods in computerized adaptive testing

Area of Science:

• Psychometrics
• Educational Measurement
• Computerized Adaptive Testing

Background:

• Test security is a critical concern in computerized adaptive testing (CAT) due to potential item information sharing among examinees.
• Stratified methods, which categorize items by their information content, are a prominent strategy to mitigate this risk.
• Current guidelines lack clarity on the optimal number of strata for item banks.

Purpose of the Study:

• To investigate the relationship between the number of strata and test security within computerized adaptive testing.
• To determine the optimal number of strata for balancing test security and measurement accuracy.
• To provide guidance on stratifying item banks to minimize item exposure.

Main Methods:

• Simulations were conducted varying the number of strata from 1 (no stratification) to the test length (maximum stratification).
• The maximum item exposure rate (r max) was manipulated across simulations.
• The trade-off between test security and measurement accuracy was analyzed by plotting their relationship.

Main Results:

• Increasing the number of strata generally enhances test security by controlling item exposure.
• Higher levels of stratification showed a positive impact on maintaining test security.
• The study found that maximizing the number of strata is the most effective strategy.

Conclusions:

• Stratifying item banks into the maximum possible number of strata is recommended for optimal test security in CAT.
• This approach effectively mitigates item exposure risks without negatively impacting measurement accuracy.
• The findings offer clear guidance for implementing stratified item selection in computerized adaptive testing.