Human-in-the-loop validation of a sequential multi-LLM medical education pipeline

Area of Science:

• Artificial Intelligence in Medicine
• Medical Education Technology
• Radiology Training

Background:

• Large language models (LLMs) offer potential for scalable medical educational content generation.
• Multi-dimensional human validation of multi-LLM pipelines for medical education has been limited.
• Radiology board preparation requires high-quality, accurate educational materials.

Purpose of the Study:

• To evaluate a sequential multi-LLM pipeline using the Gemini family for generating radiology board preparation materials.
• To assess the human validation of generated flashcards and infographics for accuracy and educational quality.
• To determine the safety and efficacy of LLM-generated medical educational content.

Main Methods:

• A 7-stage sequential multi-LLM pipeline (Gemini family) was used to produce 6000 flashcards and 833 infographics.
• Nine residents and eleven attending radiologists evaluated 1284 flashcards across 11 subspecialties.
• A two-phase evaluation design was employed, incorporating feedback after the fifth stage (S5).

Main Results:

• The evaluation-level false-negative rate for blocking errors was 1.00%, exceeding the 0.3% safety threshold.
• Fifth-stage (S5) feedback was linked to more error flags and reduced technical accuracy and educational quality scores.
• Attending radiologists identified more errors than residents in unadjusted analyses, with a gap reduction in workload-matched analyses.

Conclusions:

• Conditional safety estimates and rater-dependent patterns do not support fully automated deployment of LLM-generated medical content.
• Automated feedback sharpened evaluator scrutiny, but the study design cannot isolate specific effects.
• Further validation and refinement are necessary before widespread adoption of LLM-generated medical educational tools.