Quality Assessment of Large Language Model-Generated Medical Dialogue for Clinical Vignettes: Evaluation Study.

Area of Science:

• Medical Education Technology
• Artificial Intelligence in Healthcare
• Clinical Simulation

Background:

• Traditional clinical vignettes in medical education are limited by prototypical presentations, development time, and lack of patient diversity.
• Existing methods fail to capture patient perspectives and the dynamic nature of physician-patient interactions.
• There is a need for innovative educational tools that address these limitations.

Purpose of the Study:

• To evaluate the quality of Japanese physician-patient dialogues generated by artificial intelligence (AI).
• To assess the medical accuracy and appropriateness of AI-generated dialogues as medical interviews.
• To explore AI's potential in medical education and clinical skills training.

Main Methods:

• AI was prompted with clinical histories to generate physician-patient dialogues simulating cooperative patients.
• Dialogues covered diseases from the Japanese National Medical Licensing Examination, with 25 turns each.
• Three internists evaluated dialogues on coherence, medical accuracy (physician and patient), history content, communication, and professionalism using a 7-point Likert scale.

Main Results:

• AI-generated dialogues achieved a high overall quality score (mean composite score: 5.7/7).
• High mean scores were observed for coherence (5.9), patient medical accuracy (6.0), history taking (5.9), and communication skills (5.6).
• Essential clinical components like chief concern and diagnosis were frequently included; physical findings and test results were less consistent; treatment course was absent.

Conclusions:

• Generative AI can efficiently produce high-quality educational materials for medical training, surpassing traditional clinical vignettes.
• AI-generated dialogues offer a feasible method to practice clinical interviewing, mirroring real-world encounters.
• Physician oversight remains crucial, but AI tools can reduce time and financial burdens in medical education.