Closing the gap between open source and commercial large language models for medical evidence summarization

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Natural Language Processing

Background:

• Large language models (LLMs) show potential for summarizing medical evidence.
• Proprietary LLMs present risks like lack of transparency and vendor dependency.
• Open-source LLMs offer transparency but generally underperform proprietary models.

Purpose of the Study:

• To evaluate the impact of fine-tuning on the performance of open-source LLMs for medical evidence summarization.
• To compare the performance of fine-tuned open-source LLMs against proprietary models.

Main Methods:

• Fine-tuned three open-source LLMs (PRIMERA, LongT5, Llama-2) using the MedReview dataset (8161 systematic review-summary pairs).
• Evaluated performance through human assessment and GPT-4 simulated evaluation.
• Compared fine-tuned models against zero-shot proprietary models (e.g., GPT-3.5).

Main Results:

• Fine-tuning enhanced the performance of all tested open-source LLMs.
• Fine-tuned LongT5 achieved performance comparable to GPT-3.5 in zero-shot settings.
• Smaller fine-tuned models sometimes surpassed larger zero-shot models in summarization quality.

Conclusions:

• Fine-tuning is an effective strategy to improve open-source LLM performance for medical evidence summarization.
• Open-source LLMs, when fine-tuned, can offer a competitive and transparent alternative to proprietary models.
• This approach enhances the utility of LLMs in medical research and clinical practice.