Retrieval-Augmented Generation for Medical Question Answering on a Heart Failure Dataset: Performance Analysis

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing for Healthcare

Background:

• Retrieval-Augmented Generation (RAG) systems offer potential for improving medical question-answering (QA) systems.
• Accurate clinical support is crucial for patient care and caregiver assistance.

Purpose of the Study:

• To explore RAG framework design choices and LLM classifiers for optimizing medical QA systems.
• To enhance response quality for patient and caregiver queries across varying risk levels.

Main Methods:

• Curated a heart failure (HF) dataset with 109 questions categorized by answerability.
• Applied a RAG architecture with a structured query taxonomy and LLM classifiers.
• Evaluated retrieval and generation stages using metrics like ROUGE, BERTScore, and Intersection over Union.

Main Results:

• LLM classifier achieved 65% accuracy for answerable/deferral queries and 100% for nonanswerable queries.
• BioMedical Contrastive Pre-trained Transformers (MedCPT) cross-encoder demonstrated strong retrieval performance (93% recall @ 7).
• Despite minor reductions in ROUGE and BERT scores, Intersection over Union increased by 24%, indicating improved response accuracy.

Conclusions:

• Structured RAG with LLM classifiers enhances medical QA systems and clinical decision support.
• Systematic analysis provides guidance on optimal design choices for maximizing retrieval and response accuracy.
• Findings inform the development of robust and scalable medical QA systems.