Me-LLaMA: Medical Foundation Large Language Models for Comprehensive Text Analysis and Beyond

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing
• Large Language Models (LLMs)

Background:

• General-domain large language models (LLMs) lack specialized medical knowledge, limiting their effectiveness in healthcare.
• Existing medical LLMs often struggle to balance domain-specific knowledge with robust instruction-following capabilities.

Purpose of the Study:

• To develop Me-LLaMA, a novel family of open-source medical LLMs.
• To enhance LLM performance in medical text analysis and clinical case diagnosis by integrating specialized knowledge and instruction tuning.

Main Methods:

• Developed Me-LLaMA foundation and chat-enhanced models (13B and 70B) by continually pretraining and instruction tuning LLaMA2 models.
• Utilized a comprehensive medical dataset: 129B pre-training tokens and 214K instruction tuning samples from biomedical literature and clinical notes.
• Evaluated Me-LLaMA on six medical text analysis tasks across 12 benchmark datasets and complex clinical case diagnosis.

Main Results:

• Me-LLaMA models outperformed LLaMA and other open-source medical LLMs in zero-shot and supervised learning settings.
• Task-specific instruction-tuned Me-LLaMA surpassed leading commercial LLMs, outperforming ChatGPT and GPT-4 on multiple datasets.
• Me-LLaMA demonstrated performance comparable to ChatGPT and GPT-4 in diagnosing complex clinical cases.

Conclusions:

• Combining domain-specific continual pretraining with instruction tuning is crucial for developing effective medical LLMs.
• Me-LLaMA significantly enhances performance across diverse medical text analysis tasks and clinical applications.
• Public release of Me-LLaMA aims to foster innovation and advance medical AI for researchers and practitioners.