PMC-LLaMA: toward building open-source language models for medicine.

Area of Science:

• Artificial Intelligence
• Biomedical Informatics
• Natural Language Processing

Background:

• Large language models (LLMs) show promise but lack domain-specific knowledge for medical applications.
• General LLMs struggle with precision in specialized fields like medicine.
• There is a need for open-source, medically-tuned LLMs.

Purpose of the Study:

• To develop PMC-LLaMA, a powerful, open-source large language model tailored for medical applications.
• To enhance LLM performance in medical domains through domain-specific knowledge injection and fine-tuning.
• To provide a foundational, trainable generative language backbone for medical research.

Main Methods:

• Adapted a general-purpose LLM using data-centric knowledge injection.
• Integrated 4.8 million biomedical academic papers and 30,000 medical textbooks.
• Conducted domain-specific instruction fine-tuning on medical QA, reasoning, and dialogues using 202 million tokens.

Main Results:

• The lightweight PMC-LLaMA (13B parameters) demonstrated superior performance on medical QA benchmarks.
• PMC-LLaMA surpassed ChatGPT in evaluated medical question-answering tasks.
• All models, code, and datasets will be released to the research community.

Conclusions:

• PMC-LLaMA is an open-source LLM specifically built for the medical domain.
• Ablation studies confirmed the effectiveness of training data and model scale in medical LLMs.
• A large-scale, comprehensive dataset for instruction tuning was contributed to the research community.