Small language models learn enhanced reasoning skills from medical textbooks

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Natural Language Processing

Background:

• Large language models (LLMs) face privacy and hardware challenges in medical applications.
• Small language models (SLMs) have limited reasoning for complex medical tasks due to fewer parameters.
• Existing SLMs struggle with multi-step reasoning essential for advanced medical applications.

Purpose of the Study:

• Introduce Meerkat, a novel family of lightweight medical SLMs.
• Enhance the reasoning capabilities of SLMs for complex medical tasks.
• Address the limitations of current SLMs in medical applications.

Main Methods:

• Developed an effective training methodology for medical SLMs.
• Extracted chain-of-thought reasoning from 18 medical textbooks.
• Combined textbook reasoning with 441K medical instruction-following examples for fine-tuning.
• Fine-tuned open-source SLMs using the curated dataset.

Main Results:

• Meerkat-7B and Meerkat-8B models showed significant performance gains (22.3% and 10.6%) over counterparts on six medical exam datasets.
• Improved NEJM Case Challenge scores from 7 to 16 (Meerkat-7B) and 13 to 20 (Meerkat-8B), surpassing the human average of 13.7.
• Expert evaluations confirmed Meerkat's superiority in completeness, factuality, clarity, and logical consistency of reasoning.

Conclusions:

• Meerkat represents a significant advancement in medical SLMs, balancing lightweight design with enhanced reasoning.
• The proposed training method effectively improves SLM performance on complex medical tasks.
• Meerkat models demonstrate strong potential for practical medical applications requiring robust reasoning capabilities.