Assessment and Integration of Large Language Models for Automated Electronic Health Record Documentation in Emergency Medical Services
View abstract on PubMed
Summary
This summary is machine-generated.Integrating multiple Large Language Models (LLMs) improves accuracy in automating Electronic Health Records (EHR) documentation for emergency medical services (EMS). This approach enhances data extraction from transcribed conversations, reducing provider burden.
Area Of Science
- Medical Informatics
- Natural Language Processing
- Artificial Intelligence
Background
- Automating Electronic Health Records (EHR) documentation is vital for reducing clinician burden, especially in fast-paced emergency medical services (EMS).
- Current Natural Language Processing (NLP) methods face challenges with medical terminology, ambiguity, and numerical errors in converting transcribed speech to structured EHR data.
- Accurate and efficient EHR documentation is crucial for timely patient care and medical record integrity.
Purpose Of The Study
- To investigate the efficacy of integrating multiple Large Language Models (LLMs) for enhancing the accuracy of EMS EHR documentation.
- To evaluate the performance of an LLM integration framework compared to individual state-of-the-art LLMs in extracting structured data from transcribed EMS conversations.
- To assess the potential of this automated approach in reducing documentation workload for healthcare providers.
Main Methods
- Developed an LLM integration framework to process transcribed EMS conversations.
- Evaluated four leading LLMs (Claude 3.5, GPT-4, Gemini, Mistral) using zero-shot and few-shot learning paradigms.
- Assessed performance based on precision, recall, and F1 scores using a dataset from 40 EMS training simulations.
- Conducted a preliminary user study with domain experts to gauge perceived usefulness and identify challenges.
Main Results
- The integrated LLM framework demonstrated superior performance over individual models, achieving F1 scores of 0.78 (zero-shot) and 0.81 (few-shot).
- Quantitative evaluations showed significant improvements in structured data extraction accuracy.
- User feedback indicated potential for reduced documentation effort but highlighted issues with medical context misinterpretation and data omissions.
Conclusions
- Integrating multiple LLMs offers a promising approach to improve the accuracy and efficiency of automated EMS EHR documentation.
- The developed framework shows potential to alleviate the documentation burden on care providers.
- Further refinement is necessary to address contextual understanding and data completeness challenges for real-world implementation.