AI's Accuracy in Extracting Learning Experiences From Clinical Practice Logs: Observational Study

Area of Science:

• Medical Education Technology
• Artificial Intelligence in Healthcare
• Clinical Learning Analytics

Background:

• Improving clinical education requires understanding student experiences, but manual analysis is burdensome.
• Automated analysis of learning records and visualization of experiences can enable real-time progress tracking.
• Large language models (LLMs) show potential for analyzing clinical learning data, but their accuracy needs evaluation.

Purpose of the Study:

• To assess the accuracy of LLMs in predicting medical students' actual clinical experiences from learning log data.
• To explore the utility of LLMs for real-time progress tracking in clinical clerkships.

Main Methods:

• Learning log data from medical students at Nagoya University School of Medicine were analyzed.
• OpenAI's ChatGPT, specifically GPT-4-turbo, was used to extract experiences based on the Model Core Curriculum for Medical Education.
• A web application was developed to automate the extraction process, with accuracy evaluated against student-provided corrected lists.

Main Results:

• The study involved 20 sixth-year medical students, yielding 40 datasets.
• GPT-4-turbo demonstrated high overall specificity (99.34%) but moderate sensitivity (62.39%) in predicting clinical experiences.
• Performance varied by category, with lower sensitivity for symptoms (45.43%) and examinations (46.76%) compared to procedures (56.36%).

Conclusions:

• LLMs like GPT-4-turbo can predict clinical experiences from learning logs with high specificity and moderate sensitivity.
• Future enhancements may involve improved AI models, feedback mechanisms for learning logs, and integration with electronic medical records.
• AI-driven analysis of learning logs can potentially reduce assessment burdens and improve the quality of medical education.