A Language Model-Powered Simulated Patient With Automated Feedback for History Taking: Prospective Study

Area of Science:

• Medical Education
• Artificial Intelligence in Healthcare
• Clinical Skills Training

Background:

• History taking is crucial for medical diagnosis but challenging to teach and provide feedback on due to resource limitations.
• Virtual simulated patients and AI-powered chatbots offer innovative solutions for medical training.
• Large Language Models (LLMs) are advancing the realism and feedback capabilities of these educational tools.

Purpose of the Study:

• To evaluate the effectiveness of a Generative Pretrained Transformer (GPT) 4 model in delivering structured feedback on medical students' history-taking skills.
• To assess the quality and reliability of AI-generated feedback compared to human assessment.

Main Methods:

• A prospective study was conducted with medical students interacting with a GPT-4 powered chatbot simulating patients.
• The chatbot provided immediate feedback on the comprehensiveness of history-taking.
• Student-chatbot interactions were analyzed, comparing GPT-4 feedback with human rater feedback for interrater reliability and feedback quality.

Main Results:

• GPT-4's role-play and responses were medically plausible in over 99% of cases.
• High interrater reliability (Cohen κ=0.832) was observed between GPT-4 and human raters.
• Minor disagreements (κ<0.6) occurred in 8 categories, indicating potential issues with feedback specificity or divergence from human judgment.

Conclusions:

• The GPT-4 model demonstrated effectiveness in providing structured feedback for medical students' history-taking dialogues.
• While some limitations in feedback specificity were noted, the high agreement with human raters supports LLMs as valuable tools in medical education.
• The study advocates for the careful integration of AI-driven feedback mechanisms in medical training, highlighting key considerations for LLM implementation.