BioMistral-NLU: Towards More Generalizable Medical Language Understanding through Instruction Tuning

Area of Science:

• Artificial Intelligence
• Computational Linguistics
• Biomedical Informatics

Background:

• Large language models (LLMs) show broad generalization but struggle with specialized medical natural language understanding (NLU).
• Medical NLU requires domain knowledge, detailed text comprehension, and structured data extraction, areas where general LLMs are deficient.

Purpose of the Study:

• To develop a generalizable medical NLU model by fine-tuning an LLM on a curated medical instruction-tuning dataset.
• To improve LLM performance on specialized medical NLU tasks through a unified prompting strategy and diverse instruction tuning.

Main Methods:

• Proposed a unified prompting format for 7 NLU tasks.
• Curated MNLU-Instruct, a medical NLU instruction-tuning dataset from open-source corpora.
• Fine-tuned the BioMistral LLM on MNLU-Instruct to create the BioMistral-NLU model.

Main Results:

• BioMistral-NLU outperformed the base BioMistral, ChatGPT, and GPT-4 in zero-shot evaluations on BLUE and BLURB benchmarks.
• Instruction tuning on a diverse range of NLU tasks enhanced zero-shot generalization.
• A dataset-agnostic prompting strategy improved cross-task performance.

Conclusions:

• The proposed BioMistral-NLU model effectively bridges the gap for LLMs in specialized medical NLU tasks.
• Instruction tuning on diverse medical NLU tasks significantly enhances LLM generalizability.
• The methodology offers a promising approach for developing domain-specific LLMs.