Towards evaluating and building versatile large language models for medicine.

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Natural Language Processing

Background:

• Large language models (LLMs) show promise in various fields but require specialized evaluation for clinical applications.
• Existing benchmarks may not adequately capture the complexity and nuances of medical language understanding and reasoning.

Purpose of the Study:

• To introduce MedS-Bench, a novel benchmark for evaluating LLMs in clinical settings across 11 diverse tasks.
• To develop MedS-Ins, a large-scale instruction-tuning dataset to enhance LLM performance in medicine.
• To establish a dynamic leaderboard for tracking advancements in medical LLMs.

Main Methods:

• Evaluated nine leading LLMs on MedS-Bench, identifying performance gaps in complex clinical tasks.
• Constructed MedS-Ins, a dataset with 5 million instances from 58 medical corpora, covering 122 tasks.
• Performed instruction tuning on an open-source medical LLM using MedS-Ins, creating MMedIns-Llama 3.

Main Results:

• Most evaluated LLMs demonstrated limitations in handling complex clinical tasks within MedS-Bench.
• The MMedIns-Llama 3 model, fine-tuned on MedS-Ins, achieved superior performance on multiple clinical benchmarks.
• MedS-Ins comprises 19,000 instructions, offering a rich resource for medical LLM training.

Conclusions:

• MedS-Bench provides a robust framework for assessing clinical LLM capabilities.
• MedS-Ins significantly improves LLM performance in medical contexts, addressing current limitations.
• The open accessibility of MedS-Ins and the MedS-Bench leaderboard aim to foster collaborative progress in medical AI.