Semantic Clinical Artificial Intelligence vs Native Large Language Model Performance on the USMLE.

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing
• Medical Education Technology

Background:

• Large language models (LLMs) are increasingly used in healthcare, but require enhanced accuracy and methods to maintain performance over time.
• Evaluating LLM performance on standardized medical exams is crucial for assessing their clinical readiness.

Purpose of the Study:

• To determine if incorporating formally represented semantic clinical knowledge can improve LLM performance on the US Medical Licensing Examination (USMLE).

Main Methods:

• A comparative effectiveness research study evaluated three Llama LLMs (13B, 70B, 405B) with and without Semantic Clinical Artificial Intelligence (SCAI) retrieval augmented generation (RAG).
• LLM performance was assessed using text-based questions from USMLE Steps 1, 2, and 3, with accuracy determined against the official answer key.

Main Results:

• SCAI RAG significantly improved LLM performance on USMLE Steps 1, 2, and 3.
• The 13B LLM met the passing threshold on Step 3 with SCAI RAG (60.2%).
• The 70B and 405B LLMs passed all steps with or without SCAI RAG, with the 70B model achieving 92.0% on Step 1 and the 405B model achieving 95.1% on Step 3.

Conclusions:

• Semantic clinical knowledge integration via SCAI RAG enhances LLM accuracy for medical licensing examinations.
• LLMs augmented with targeted clinical knowledge show promise for improving medical question-answering capabilities and facilitating healthcare implementation.
• Semantic reasoning represents a key advancement for improving LLM performance in critical medical applications.