Retrieval-Augmented Generation for Medical Decision-Making in Emergency Care

Medical decision-making in emergency care frequently relies on the integration of extensive amounts of guideline-based knowledge, which poses challenges for timely and accurate decisions. This study evaluated a retrieval-augmented generation (RAG) system, with a focus on cardiology and gastroenterology, by comparing semantic and sparse retrieval mechanisms with the Mixtral-8x7B Instruct large language model (LLM) across 100 German clinical cases. Semantic retrieval exhibited superior performance in identifying relevant guidelines and demonstrated enhanced temporal performance. However, while the semantic retrieval enhanced the guideline selection process, physicians did not rate its responses as superior in terms of medical and content accuracy, case relevance, factual integrity, and overall usability when compared to those from the Mixtral model without a retrieval mechanism. In addition, the use of retrieval mechanisms often resulted in answers with linguistic inconsistencies. This performance deficit and error susceptibility of the RAG system was predominantly attributable to the constraints inherent in the automated splitting of chunks. Future improvements are slated to include preprocessing guidelines, refinement of retrieval methods, and leveraging more advanced LLMs.Clinical Relevance- RAG systems can provide accurate, guideline-based recommendations for medical decision-making in emergency care, which could increase efficiency and reduce diagnostic errors in healthcare.