Evaluating Retrieval Augmented Generation-enhanced Large Language Models for Question Answering On German...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Clinical Guideline Application

Background:

• Large Language Models (LLMs) show promise for medical information retrieval.
• Accurate question answering based on complex medical guidelines remains a challenge.
• Retrieval-augmented Generation (RAG) offers a potential solution to enhance LLM performance.

Purpose of the Study:

• To evaluate the effectiveness of RAG-enhanced LLMs in answering questions related to German neurovascular guidelines.
• To compare the performance of different LLMs with and without RAG integration.
• To analyze the retrieval performance of various strategies for medical guideline information.

Main Methods:

• Four LLMs (GPT-4o-mini, Llama 3.1, Mixtral 8×22B, Claude 3.5 Sonnet) were tested with RAG.
• GPT-4o-mini without RAG served as a baseline.
• Answers were expert-validated, and retrieval strategies were assessed using a synthetic dataset.

Main Results:

• Claude 3.5 Sonnet demonstrated the highest accuracy (70.6%) among RAG-enhanced LLMs.
• RAG significantly improved LLM performance compared to models without RAG (20.0% accuracy).
• Retrieval errors constituted 80% of incorrect answers, with BM25 outperforming vector-based methods.

Conclusions:

• RAG substantially boosts LLM accuracy for medical guideline Q&A, though error rates persist.
• Further improvements in accuracy and confidence metrics are crucial for clinical deployment.
• General LLMs exhibit strong performance in non-English medical Q&A without specialized training.