Related Experiment Video
Updated: Aug 13, 2026
03:14
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Evaluating Retrieval-Augmented Large Language Models on Anesthesiology Board-Style Questions: Benchmark Study
Nguyen Quang Phuong1, Shanq-Jang Ruan1, Pei-Fu Chen2
1Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan.
JMIR Formative Research
|August 11, 2026
Summary
Retrieval-augmented generation (RAG) improves large language model (LLM) performance on anesthesiology exams, with Qwen reasoning models outperforming others. Optimized RAG configurations and semantic chunking enhance accuracy for medical education applications.
Area of Science:
- Medical Education Technology
- Artificial Intelligence in Healthcare
- Natural Language Processing
Background:
- Open-source, mid-scale large language models (LLMs) offer privacy-preserving alternatives to ultra-large models in healthcare.
- Retrieval-augmented generation (RAG) enables smaller LLMs to tackle specialized medical domains like anesthesiology.
- Limited studies evaluate RAG on complex medical examinations, necessitating rigorous benchmarking.
Purpose of the Study:
- Systematically evaluate RAG pipelines for anesthesiology board-style questions.
- Quantify the impact of design choices (hyperparameters, embeddings, chunking) on RAG performance.
- Compare reasoning-oriented LLMs against conventional LLMs within RAG frameworks.
Main Methods:
- Large-scale benchmarking using American Board of Anesthesiology-style multiple-choice questions.
- Optimization on a diagnostic set, followed by validation on a larger corpus.
- Evaluation of various LLMs (Llama, Qwen) and RAG configurations, including self-reflective RAG (self-RAG).
Main Results:
- RAG frameworks increased correct answers, with optimal stability at low temperature and nucleus sampling.
- High-capacity embeddings and semantic chunking significantly improved accuracy.
- Qwen reasoning models, particularly Qwen-3-32B, outperformed larger conventional LLMs, demonstrating robustness.
Conclusions:
- RAG systems enhance performance on anesthesiology board questions, contingent on retrieval design.
- Optimized retrieval, embeddings, and chunking strategies improve LLM robustness and accuracy.
- Reasoning-oriented models show potential to compensate for smaller parameter size in specific applications.