Answering Patterns in SBA Items: Students, GPT3.5, and Gemini.

Area of Science:

• Medical Education
• Artificial Intelligence in Assessment
• Cognitive Science

Background:

• Large language models (LLMs) are increasingly utilized for generating and answering educational assessments.
• Limited research exists on comparing LLM performance across multiple iterations using item statistics, particularly in medical education.

Purpose of the Study:

• To investigate the answering patterns of LLMs (GPT-3.5, Gemini) on single-best answer (SBA) questions.
• To compare LLM performance and answering patterns against first-year medical students.
• To evaluate the suitability of free-to-use LLMs for medical education assessment.

Main Methods:

• Forty-one SBA questions designed for first-year medical students were used.
• GPT-3.5 and Gemini were tested across 100 iterations each.
• LLM responses were analyzed for answering patterns and compared with student performance data.

Main Results:

• LLMs demonstrated more repetitive and clustered answering patterns than students.
• Students outperformed LLMs in managing multiple-choice options within the SBA format.
• Free LLMs were found to be inferior to well-trained students or specialists for technical medical questions.

Conclusions:

• LLMs exhibit problematic repetitive answering, potentially compounding errors.
• Current free LLMs struggle with contextual interpretation in technical medical assessments.
• Human oversight remains critical in medical education assessment processes involving AI.