An Experimental Design to Investigate Item Parameter Drift.

Area of Science:

• Psychometrics
• Educational Measurement
• Statistical Modeling

Background:

• Traditional methods for detecting item parameter drift are insufficient when all items are exposed.
• Item parameter drift can significantly impact test validity and fairness.
• High-stakes exams require robust methods to ensure score integrity.

Purpose of the Study:

• To propose and illustrate a novel strategy for detecting item parameter drift using only unexposed items.
• To address the limitations of existing drift detection methods in fully exposed item scenarios.
• To evaluate the effectiveness of the proposed method in a real-world high-stakes testing context.

Main Methods:

• A stratified random sampling method was employed to deploy unexposed items within an experimental design.
• The strategy focuses on analyzing data from a subset of unexposed items to infer potential drift in exposed items.
• The method was applied to investigate unexpected score increases observed on a licensure examination.

Main Results:

• The application of the proposed method to the licensure exam data yielded results suggestive of item parameter drift.
• Observed score increases indicated a potential shift in item characteristics.
• Statistical significance was not reached at the conventional 0.05 alpha level, indicating inconclusive findings.

Conclusions:

• The proposed strategy offers a viable approach for detecting item parameter drift, particularly when all items are at risk.
• Further research and validation are needed to refine the method and confirm its sensitivity.
• The findings underscore the importance of continuous monitoring for item parameter drift in standardized testing.