A Graphical Method for Displaying the Model Fit of Item Response Theory Trace Lines

Area of Science:

• Educational Measurement
• Psychometrics
• Statistical Modeling

Background:

• Item response theory (IRT) is a statistical framework for educational test development and student assessment.
• The three-parameter logistic model is widely used in educational testing due to its flexibility.
• A graphical method for examining model fit for this specific IRT model is currently lacking.

Purpose of the Study:

• To introduce a novel graphical method, the "bin plot," for assessing the fit of the three-parameter logistic IRT model.
• To provide an IRT-specific graphical tool analogous to scatterplots used in linear regression.

Main Methods:

• Developed the "bin plot" which displays IRT trace lines against binned student performance data.
• Binned students based on their overall test performance.
• Plotted the proportion of students in each bin who answered a focal question correctly against the trace line.
• Utilized computer simulations to evaluate different binning strategies and the performance of confidence intervals.

Main Results:

• Binning students based on the total number of correctly answered questions was found to be the most effective strategy.
• Confidence intervals for bin plots demonstrated coverage probabilities close to nominal values in typical testing situations.
• Certain scenarios were identified where confidence intervals exhibited inflated error rates.

Conclusions:

• The "bin plot" offers a valuable visual tool for evaluating the fit of the three-parameter logistic IRT model.
• The proposed binning strategy and confidence intervals provide a reliable method for model fit assessment.
• Further research may be needed to address scenarios where confidence intervals show inflated error rates.