Performance Evaluation of Large Language Models With Retrieval-Augmented Generation in Cardiology Specialist...

Area of Science:

• Artificial Intelligence in Medicine
• Medical Education Technology
• Cardiology Training

Background:

• Large language models (LLMs) show potential in medical education.
• Application of LLMs to cardiology specialist examinations is underexplored.
• This study compares a specialized RAG-LLM (CardioCanon) against general-purpose LLMs.

Purpose of the Study:

• To evaluate the performance of a retrieval-augmented generation LLM (RAG-LLM) named CardioCanon.
• To compare CardioCanon's accuracy against general-purpose LLMs on cardiology specialist examination questions.
• To assess the impact of RAG techniques on AI-assisted examination performance.

Main Methods:

• Utilized 96 publicly available, text-based, open-source multiple-choice questions from the Japanese Cardiology Specialist Examination (1997-2022).
• Compared the performance of CardioCanon against ChatGPT-4o and Gemini 2.0 Flash.
• Evaluated both option-level and case-based accuracy.

Main Results:

• CardioCanon achieved similar option-level accuracy to ChatGPT-4o (81.0% vs. 76.0%) and Gemini 2.0 Flash (81.0% vs. 77.2%).
• CardioCanon demonstrated significantly higher case-based accuracy than ChatGPT-4o (57.3% vs. 29.2%, P<0.001).

Conclusions:

• Retrieval-augmented generation (RAG) techniques can enhance AI-assisted examination performance.
• RAG improves case-level reasoning and decision-making in AI models for medical education.
• CardioCanon shows potential for improving cardiology specialist training and assessment.