Leveraging large language models to construct feedback from medical multiple-choice Questions.

Area of Science:

• Medical Education
• Artificial Intelligence in Healthcare

Background:

• Formative assessments, such as the Progress Test Medizin (PTM), can be improved by providing feedback beyond numerical scores.
• Content-based feedback, derived from exam questions, offers students insights into performance and aids revision.

Purpose of the Study:

• To explore the utility of Large Language Models (LLMs) in generating content-based feedback for the PTM.
• To comparatively assess the effectiveness of two LLMs in this task.
• To gauge medical practitioners' and educators' perceptions of LLM-generated feedback for the PTM.

Main Methods:

• Utilized two popular LLMs to generate content-based feedback for the PTM.
• Conducted a comparative assessment of LLM outputs using textual similarity.
• Administered a survey to medical practitioners and educators regarding LLM feedback utility.

Main Results:

• Both LLMs demonstrated similar performance, with individual strengths and weaknesses.
• One LLM, a paid service, produced slightly superior outputs compared to the free alternative.
• Survey participants found the LLM-generated feedback relevant, useful, and expressed openness to future use.

Conclusions:

• LLM-generated content-based feedback can be a valuable addition to the numerical feedback currently provided in the PTM.
• Despite imperfections, LLMs show promise in enhancing medical education assessment tools.
• Medical professionals are receptive to integrating LLMs into educational feedback mechanisms.