Field-Testing Multiple-Choice Questions With AI Examinees: English Grammar Items

Area of Science:

• Educational Measurement
• Artificial Intelligence in Education
• Psychometrics

Background:

• Field-testing educational assessments is crucial but resource-intensive.
• Current methods rely on human examinees, leading to significant time and cost investments.
• The development of high-quality assessments necessitates efficient and effective field-testing procedures.

Purpose of the Study:

• To introduce and evaluate an innovative method for field-testing educational assessment items using artificially intelligent (AI) examinees.
• To demonstrate the feasibility of using AI examinees to generate item response data comparable to human examinees.
• To explore the potential of AI in streamlining the assessment development lifecycle and reducing associated costs.

Main Methods:

• Fine-tuning pre-trained transformer language models based on the 2-parameter logistic (2PL) item response model to simulate human test-taker behavior.
• Utilizing AI examinees, each assigned a latent ability (θ), to predict response selection probabilities for multiple-choice English grammar questions.
• Assessing the performance of the AI examinee model by comparing true and predicted 2PL correct response probabilities.

Main Results:

• The best AI modeling approach achieved a correlation of .82 between true and predicted 2PL correct response probabilities (bias = 0.00, RMSE = 0.18).
• AI-generated item response data demonstrated utility for calculating item proportion correct, item discrimination, conducting item calibration with anchors, distractor analysis, dimensionality analysis, and latent trait scoring.
• The AI approach did not reach the accuracy levels of human examinee response data.

Conclusions:

• AI examinees offer a viable and promising alternative for field-testing educational assessments, capable of generating data for various psychometric analyses.
• While current AI models do not fully replicate human examinee accuracy, the potential for substantial resource savings in terms of time, cost, and logistical concerns is enormous.
• Further refinement of AI models could revolutionize the field-testing process, enabling faster assessment development, improved item bank expansion, and enhanced test security.