Medical LLMs: Fine-Tuning vs. Retrieval-Augmented Generation

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Natural Language Processing

Background:

• Large language models (LLMs) possess broad knowledge but lack specialized expertise.
• Domain-specific LLM performance can be enhanced through targeted training methodologies.
• Healthcare applications require specialized LLMs capable of accurate and reliable information retrieval and generation.

Purpose of the Study:

• To evaluate the effectiveness of retrieval-augmented generation (RAG) and fine-tuning (FT) on healthcare data for five distinct large language models.
• To compare the performance of RAG alone, FT alone, and a combined FT+RAG approach.
• To identify which models and training strategies yield superior results for specialized healthcare applications.

Main Methods:

• Five large language models (LLama-3.1-8B, Gemma-2-9B, Mistral-7B-Instruct, Qwen2.5-7B, Phi-3.5-Mini-Instruct) were fine-tuned on the MedQuAD dataset.
• Models were trained using three approaches: retrieval-augmented generation (RAG) only, fine-tuning (FT) only, and a combination of both (FT+RAG).
• Performance was assessed across multiple metrics to evaluate domain-specific accuracy and capabilities.

Main Results:

• Retrieval-augmented generation (RAG) and combined fine-tuning (FT+RAG) consistently outperformed fine-tuning (FT) alone across most evaluated models.
• LLAMA and PHI models demonstrated superior performance, with LLAMA showing overall excellence and PHI excelling in RAG/FT+RAG capabilities.
• QWEN models generally lagged in performance, while GEMMA and MISTRAL exhibited varied results depending on the training approach.

Conclusions:

• Specialized training methods, particularly RAG and FT+RAG, are crucial for enhancing LLM performance in niche domains like healthcare.
• The choice of LLM architecture significantly impacts the effectiveness of different training strategies.
• Further research into optimizing these training methods can unlock greater potential for AI in medical information systems.