The Asymptotic Distribution of Average Test Overlap Rate in Computerized Adaptive Testing

Area of Science:

• Psychometrics
• Educational Measurement
• Computerized Adaptive Testing (CAT)

Background:

• The average test overlap rate is a key metric for assessing computerized adaptive test (CAT) security and item pool usage.
• Current understanding lacks a formal statistical distribution for this crucial measure, hindering rigorous analysis.

Purpose of the Study:

• To derive the asymptotic sampling distribution of the average test overlap rate in fixed-length CAT.
• To enable the estimation of standard errors and the construction of confidence intervals for this statistic.
• To facilitate statistical comparisons between different CAT designs based on their overlap rates.

Main Methods:

• Mathematical derivation of the asymptotic distribution for a linear transformation of the average test overlap rate.
• A simulation study to compare average test overlap rates between two CAT designs with varying exposure control strategies.

Main Results:

• The asymptotic distribution of the average test overlap rate in fixed-length CAT has been established.
• Theoretical results allow for the calculation of standard errors and confidence intervals.
• Simulation results provide a practical demonstration of comparing overlap rates across different CAT designs.

Conclusions:

• The derived sampling distribution provides a robust statistical foundation for analyzing average test overlap rates in CAT.
• This advancement supports more accurate assessments of test security and item pool utilization.
• The findings facilitate evidence-based decisions in the design and implementation of CAT systems.