Intelligent documentation in medical education: can AI replace manual case logging?

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Radiology Training

Background:

• Procedural case log documentation is a critical but time-consuming task for radiology residents.
• Manual logging of procedures requires significant administrative effort, impacting training efficiency.
• Current methods for case log documentation are prone to errors and inefficiencies.

Purpose of the Study:

• To evaluate the feasibility of using large language models (LLMs) for automated procedural case log documentation in radiology.
• To assess the accuracy and efficiency of LLMs compared to manual logging.
• To identify challenges in procedure type extraction and integration into clinical workflows.

Main Methods:

• Retrospective analysis of 414 radiology reports from nine interventional radiology residents (2018-2024).
• Testing of local (Qwen-2.5) and commercial (Claude-3.5) LLMs using instruction and chain-of-thought prompting.
• Performance evaluation based on sensitivity, specificity, F1-score, inference time, and token efficiency.

Main Results:

• Both local and commercial LLMs outperformed the standard benchmark.
• Qwen-2.5 achieved an F1-score of 86.66% with chain-of-thought prompting.
• Claude-3.5-Haiku reached an F1-score of 86.89% with sub-2s latency.
• Automation could save over 35 hours of manual annotation per resident annually.
• Local LLM deployment offered lower recurring costs, while commercial models provided faster inference.

Conclusions:

• LLMs offer a scalable and accurate solution for automating radiology case log documentation.
• Optimizing for procedure-specific challenges and seamless system integration are crucial for adoption.
• Further validation across multi-institution datasets and exploration of prompting strategies are recommended.