RAGCare-QA: A benchmark dataset for evaluating retrieval-augmented generation pipelines in theoretical medical...

Area of Science:

• Medical Education
• Artificial Intelligence in Healthcare
• Natural Language Processing

Background:

• Assessing the efficacy of Retrieval-Augmented Generation (RAG) in medical education requires specialized datasets.
• Existing resources may not adequately cover the theoretical knowledge crucial for medical training and evaluation.

Purpose of the Study:

• To introduce RAGCare-QA, a comprehensive dataset designed for evaluating RAG pipelines in medical education.
• To provide a standardized tool for assessing the performance of AI-driven medical knowledge systems.

Main Methods:

• Developed RAGCare-QA, a dataset comprising 420 theoretical medical knowledge questions.
• Questions cover six medical specialties (Cardiology, Endocrinology, Gastroenterology, Family Medicine, Oncology, Neurology).
• Included questions with three complexity levels (Basic, Intermediate, Advanced) and mapped them to RAG implementation types (Basic, Multi-vector, Graph-enhanced).

Main Results:

• The dataset contains 420 one-choice-only questions.
• 75.0% of questions are suited for Basic RAG, 19.5% for Multi-vector RAG, and 5.5% for Graph-enhanced RAG.
• Questions focus on fundamental concepts, pathophysiology, diagnostic criteria, and treatment principles.

Conclusions:

• RAGCare-QA serves as a valuable resource for assessing RAG-based medical education systems.
• The dataset enables researchers to fine-tune retrieval methods for diverse theoretical medical knowledge questions.
• Facilitates the advancement of AI applications in medical training and evaluation.