A comprehensive evaluation of large Language models on benchmark biomedical text processing tasks.

Area of Science:

• Biomedical Informatics
• Artificial Intelligence

Background:

• Large Language Models (LLMs) have shown broad task applicability.
• Their performance in the specialized biomedical domain remains largely unexplored.

Purpose of the Study:

• To comprehensively evaluate and compare the capabilities of popular LLMs on diverse biomedical tasks.
• To assess LLM performance against established models, particularly in data-scarce scenarios.

Main Methods:

• Conducted a comprehensive evaluation of 4 LLMs across 6 biomedical tasks and 26 datasets.
• Compared zero-shot LLM performance against fine-tuned state-of-the-art models on smaller datasets.

Main Results:

• LLMs demonstrate strong performance in zero-shot settings on small biomedical datasets, sometimes exceeding fine-tuned models.
• No single LLM consistently outperformed others across all tasks; performance was task-dependent.
• Overall LLM performance lagged behind models fine-tuned on extensive datasets.

Conclusions:

• Pre-training on large corpora equips LLMs with significant biomedical domain specialization.
• LLMs present a valuable potential tool for biomedical applications with limited annotated data.
• Further research is needed to optimize LLM performance for complex biomedical challenges.