Monitoring scale scores over time via quality control charts, model-based approaches, and time series techniques

Area of Science:

• Educational Measurement
• Psychometrics
• Statistical Quality Control

Background:

• Maintaining score scale stability is crucial for standardized educational assessments.
• Traditional quality control methods for scale drift are often time-consuming or require specific equating designs, hindering timely detection of issues.
• Existing methods are insufficient for continuous monitoring and rapid identification of unusual testing outcomes.

Purpose of the Study:

• To present a novel approach for continuous score monitoring and assessment of scale drift in standardized educational tests.
• To address the limitations of traditional methods in timely detection of score scale anomalies.
• To develop a method that accommodates continuous monitoring, adjustment for variations, shift identification, and autocorrelation assessment.

Main Methods:

• Implementation of quality control charts for monitoring score scales.
• Application of model-based approaches for statistical analysis.
• Utilization of time series techniques to analyze score data.
• Evaluation using manipulated data from 71 administrations of a large-scale language assessment.

Main Results:

• The proposed approach enables continuous monitoring of score scales.
• The method effectively identifies abrupt shifts and assesses autocorrelation in test scores.
• Performance evaluation demonstrated the viability of the new methodologies.

Conclusions:

• The developed approach offers a timely and efficient solution for monitoring score scale stability in educational assessments.
• This method enhances the ability to detect and address scale drift promptly.
• The findings support the regular use of these techniques in operational testing environments.