Garbage In, Garbage Out: Context Engineering for Generating Multiple-Choice Questions for Medical Education Using...

Area of Science:

• Medical Education Technology
• Artificial Intelligence in Education
• Knowledge Representation

Background:

• Automated generation of high-quality multiple-choice questions (MCQs) for medical education is a persistent challenge.
• Existing methods often struggle with contextual accuracy and relevance.

Purpose of the Study:

• To develop and evaluate KiMED, a novel platform for automated MCQ generation in German.
• To enhance MCQ quality by integrating Knowledge Graph (KG)-assisted retrieval with large language models (LLMs).

Main Methods:

• Biochemistry course materials were processed to construct a KG, extracting entities, properties, and relationships.
• A multi-agent system utilized the KG to generate MCQs, including question stems, correct answers (keys), and incorrect answers (distractors).
• LLM-based generation was enhanced by KG-assisted retrieval for precise, contextually relevant information.

Main Results:

• The constructed KG accurately represented 87% of entities and 82% of relationships from the source material.
• MCQs generated using the KG approach achieved a 45% usability rate among experts.
• MCQs generated from unstructured text showed a significantly lower usability rate of 23%.

Conclusions:

• Structured context, particularly through KGs, substantially improves the quality of automatically generated MCQs.
• Further optimization of context, agent workflows, and post-processing is necessary for reliable automated question generation.