Closing the gap between open-source and commercial large language models for medical evidence summarization

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Medical Informatics

Background:

• Large language models (LLMs) show potential for summarizing medical evidence.
• Proprietary LLMs offer high performance but lack transparency and create vendor dependency.
• Open-source LLMs provide transparency and customization but generally underperform proprietary models.

Purpose of the Study:

• To evaluate the effectiveness of fine-tuning open-source LLMs for medical evidence summarization.
• To determine if fine-tuning can bridge the performance gap between open-source and proprietary LLMs.

Main Methods:

• Utilized the MedReview benchmark dataset (8,161 systematic review-summary pairs).
• Fine-tuned three open-source LLMs: PRIMERA, LongT5, and Llama-2.
• Evaluated performance using ROUGE-L, METEOR, and CHRF scores, alongside human and GPT-4 simulated evaluations.

Main Results:

• Fine-tuned LLMs showed significant improvements: +9.89 ROUGE-L, +13.21 METEOR, +15.82 CHRF.
• Fine-tuned LongT5 approached GPT-3.5 performance in zero-shot settings.
• Smaller fine-tuned models sometimes outperformed larger zero-shot models.
• Performance gains were consistent across automated and human evaluations.

Conclusions:

• Fine-tuning substantially boosts open-source LLM performance for medical evidence summarization.
• This approach offers a viable, transparent alternative to proprietary models.
• Results guide the selection of LLMs for specialized tasks like medical evidence synthesis.