Using a Model of Analysts' Judgments to Augment an Item Calibration Process

Area of Science:

• Educational Measurement
• Psychometrics
• Data Analysis

Background:

• Item review is crucial for assessing field test (FT) item fit to item response theory (IRT) models.
• Current manual review processes are time-consuming and prone to human error, especially with large datasets.
• Behavioral decision-making research suggests parametric models can outperform human judgment.

Purpose of the Study:

• To develop a model that mimics analyst decision-making in FT item reviews.
• To create a rule-based system for classifying item status, distinguishing between clear-cut decisions and those needing human review (HR).
• To reduce the burden of manual item calibration and improve efficiency.

Main Methods:

• Utilized behavioral decision-making principles to model analyst integration of statistical and graphical FT item data.
• Developed a set of decision rules based on this model to predict item status.
• Evaluated the model's performance in classifying items requiring HR.

Main Results:

• The developed model successfully mimics analyst integration of FT item statistics and performance plots.
• The proposed decision rules achieve a desired level of classification accuracy.
• Implementation of these rules led to an estimated 65% reduction in calibrations requiring human reviews (HR).

Conclusions:

• A data-driven, model-based approach can effectively automate aspects of the FT item review process.
• This methodology significantly enhances efficiency by reducing the need for extensive human reviews (HR).
• The findings support the use of predictive models in psychometric analysis for improved workflow and accuracy.