Me-LLaMA: Foundation Large Language Models for Medical Applications

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Natural Language Processing

Background:

• Large language models (LLMs) show promise in medicine but lack specialized medical training.
• Existing open-source medical LLMs have limitations in clinical application.
• The challenge lies in adapting general LLMs for nuanced medical data.

Purpose of the Study:

• Introduce Me-LLaMA, a novel family of medical large language models (LLMs).
• Develop foundation and chat-enhanced models using LLaMA2 architecture and extensive medical datasets.
• Evaluate Me-LLaMA's performance against existing models and benchmarks.

Main Methods:

• Utilized continual pre-training on a 129B token medical dataset.
• Employed instruction tuning with a 214k sample dataset.
• Developed and used the Medical Instruction Benchmark Evaluation (MIBE) across six medical tasks and 12 datasets.

Main Results:

• Me-LLaMA models outperformed existing open-source medical LLMs in zero-shot, few-shot, and supervised learning.
• Task-specific instruction-tuned Me-LLaMA surpassed ChatGPT on 7/8 and GPT-4 on 5/8 MIBE datasets.
• Me-LLaMA demonstrated superior mitigation of the catastrophic forgetting problem.

Conclusions:

• Me-LLaMA represents a significant advancement in open-source medical LLMs, integrating biomedical and clinical data.
• The models offer enhanced performance on both general and medical tasks, making them suitable for medical AI applications.
• Me-LLaMA provides a robust foundation for future medical AI research and deployment.