Retrieval-Augmented Generation for Medical Decision-Making in Emergency Care

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Clinical Decision Support

Background:

• Emergency care decision-making requires integrating complex clinical guidelines.
• Timely and accurate decisions are challenged by the volume of medical knowledge.
• Large language models (LLMs) offer potential but require effective knowledge integration.

Purpose of the Study:

• To evaluate a retrieval-augmented generation (RAG) system for medical decision-making in emergency care.
• To compare semantic and sparse retrieval mechanisms integrated with the Mixtral-8x7B Instruct LLM.
• To assess the system's performance in cardiology and gastroenterology using German clinical cases.

Main Methods:

• A RAG system was developed, incorporating semantic and sparse retrieval.
• The system was tested on 100 German clinical cases in cardiology and gastroenterology.
• Performance was evaluated by comparing retrieval accuracy, temporal efficiency, and physician-rated response quality.

Main Results:

• Semantic retrieval outperformed sparse retrieval in identifying relevant guidelines and temporal performance.
• Physicians did not rate RAG system responses as superior in accuracy or usability compared to the base LLM.
• Linguistic inconsistencies and errors in RAG outputs were linked to automated text chunking limitations.

Conclusions:

• RAG systems can aid guideline-based recommendations in emergency care, potentially improving efficiency and reducing errors.
• Further research is needed to optimize RAG systems, focusing on guideline preprocessing, retrieval refinement, and advanced LLMs.
• Addressing limitations in text chunking is crucial for enhancing RAG system reliability and clinical utility.