AI-generated biochemistry test item parameters in MST test conditions

Area of Science:

• Medical Education Assessment
• Artificial Intelligence in Education
• Psychometrics

Background:

• Investigating the accuracy of AI tools like ChatGPT 4o for estimating medical assessment item difficulty.
• Comparing AI predictions against empirically derived parameters from multistage testing simulations.

Purpose of the Study:

• To evaluate ChatGPT 4o's capability in estimating the difficulty of medical assessment items.
• To compare AI-generated difficulty estimates with simulation-derived parameters.

Main Methods:

• Utilizing a hybrid simulation-validation design.
• Generating 80 multiple-choice biochemistry questions with AI-estimated difficulty parameters (b-parameters).
• Administering questions via simulated multistage testing to 5,000 virtual examinees.

Main Results:

• Moderate agreement (r=0.612) found between AI-generated and simulation-derived difficulty parameters.
• ChatGPT 4o systematically overestimated item difficulty (mean bias=0.240).
• AI estimates were less accurate for very easy items, with higher error rates compared to medium difficulty items.

Conclusions:

• ChatGPT 4o shows promise for preliminary item generation in medical education but requires empirical calibration and expert oversight.
• Simulation-based validation has limitations; actual student performance data is crucial for rigorous psychometric validation.
• Integrating AI into medical education assessment necessitates careful validation to address systematic biases and ensure accuracy.