Custom Large Language Models Improve Accuracy: Comparing Retrieval Augmented Generation and Artificial Intelligence...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Orthopaedic Surgery

Background:

• Large Language Models (LLMs) show promise in medicine but face skepticism due to variable accuracy.
• Standard LLMs may not consistently provide reliable medical information.
• Customization techniques are needed to enhance LLM accuracy for clinical applications.

Purpose of the Study:

• To evaluate the efficacy of Retrieval Augmented Generation (RAG) and Agentic Augmentation in improving LLM accuracy.
• To compare custom LLM methods against standard LLMs using an anterior cruciate ligament (ACL) injury case.
• To demonstrate the value of tailored LLMs for delivering precise medical information.

Main Methods:

• 100 ACL-related questions and answers based on 2022 AAOS guidelines were curated.
• Standard LLMs (closed-source and open-source) were tested, then augmented with RAG and AI agents.
• Fellowship-trained surgeons blindly assessed response accuracy; ROUGE and MEAT scores were calculated.

Main Results:

• Non-custom LLMs performed below 60% accuracy.
• RAG improved accuracy by an average of 39.7% across all models.
• Meta's Llama3 70b (RAG-only) achieved 94% accuracy; OpenAI's GPT-4 (RAG + AI agents) reached 95%.

Conclusions:

• Retrieval Augmented Generation (RAG) significantly enhances LLM accuracy in medical contexts.
• Agentic augmentation further refines LLM performance, achieving near-perfect accuracy.
• Customized LLMs, like RAG- and agent-augmented models, can serve as reliable sources for orthopaedic information, supporting patient decision-making.