Retrieval-augmented generation enhances large language model performance on the Japanese orthopedic board examination

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing
• Medical Education Technology

Background:

• Large language models (LLMs) show promise in healthcare but require domain-specific accuracy.
• Retrieval-Augmented Generation (RAG) aims to enhance LLMs by reducing hallucinations and improving access to specialized information.
• The effectiveness of RAG in specialized medical domains is not fully understood.

Purpose of the Study:

• To evaluate the impact of RAG on the performance of LLMs (GPT-3.5 Turbo, GPT-4o, o1-preview) in a specialized medical examination.
• To determine if RAG can bridge the accuracy gap for LLMs in medical applications.
• To assess the suitability of RAG-enhanced LLMs for medical use.

Main Methods:

• A specialized orthopedic database was created from a standard textbook.
• GPT-3.5 Turbo, GPT-4o, and o1-preview were tested with and without RAG on Japanese orthopedic examination questions (text and image-based).
• Error analysis was performed to identify performance determinants.

Main Results:

• GPT-3.5 Turbo showed no significant improvement with RAG (28% accuracy with RAG vs. 29% without).
• GPT-4o accuracy increased from 62% to 72% with RAG.
• o1-preview accuracy improved substantially from 67% to 84% with RAG.

Conclusions:

• RAG significantly enhanced the performance of GPT-4o and o1-preview, with o1-preview achieving clinically relevant proficiency.
• GPT-3.5 Turbo's performance did not improve with RAG, indicating a lack of necessary reasoning abilities.
• RAG integration shows potential for improving LLM accuracy in specialized medical fields, particularly for advanced models.