A Pipeline for Automating Emergency Medicine Documentation Using LLMs with Retrieval-Augmented Text Generation

Area of Science:

• Emergency Medicine
• Natural Language Processing
• Clinical Informatics

Background:

• Manual documentation in emergency settings is inefficient and error-prone.
• Large Language Models (LLMs) offer potential for improving medical communication.
• Clinical deployment of LLMs in German faces accuracy, relevance, and privacy challenges.

Purpose of the Study:

• Develop and evaluate an automated pipeline for emergency medical documentation in German.
• Generate synthetic dialogues for controlled NLP performance evaluation.
• Design a pipeline to retrieve critical clinical information from emergency dialogues.

Main Methods:

• Utilized a subset of 100 anonymized patient records from the MIMIC-IV-ED dataset.
• Employed a Retrieval-Augmented Generation (RAG) system with chunking, embedding, and dynamic prompts.
• Evaluated performance using precision, recall, F1-score, and sentiment analysis.

Main Results:

• Achieved high extraction accuracy for medication data (F1-scores: 86.21%-100%).
• Demonstrated effectiveness of the automated pipeline in retrieving key clinical features.
• Identified performance decline in nuanced clinical language, indicating areas for refinement.

Conclusions:

• The developed RAG pipeline shows promise for automated emergency medical documentation in German.
• Further refinement is needed to address challenges in extracting complex clinical information.
• This approach can enhance efficiency and accuracy in emergency healthcare documentation.