Benchmarking large language models for biomedical natural language processing applications and recommendations.

Area of Science:

• Biomedical Natural Language Processing (BioNLP)
• Artificial Intelligence in Healthcare
• Computational Linguistics

Background:

• The exponential growth of biomedical literature necessitates automated knowledge extraction.
• Biomedical Natural Language Processing (BioNLP) offers a solution for efficient information synthesis.
• The efficacy of Large Language Models (LLMs) in specialized BioNLP tasks is not well-established.

Purpose of the Study:

• To systematically evaluate the performance of leading Large Language Models (LLMs) on diverse BioNLP benchmarks.
• To compare LLM performance (zero-shot, few-shot, fine-tuning) against traditional fine-tuned models like BERT and BART.
• To identify practical challenges and provide insights for LLM application in BioNLP.

Main Methods:

• Evaluation of four LLMs (GPT, LLaMA representatives) across 12 BioNLP benchmarks and six application types.
• Comparative analysis of zero-shot, few-shot, and fine-tuning approaches for LLMs.
• Benchmarking against fine-tuned BERT and BART models, including analysis of inconsistencies, hallucinations, and cost.

Main Results:

• Traditional fine-tuned models generally outperform zero- or few-shot LLMs on most BioNLP tasks.
• Closed-source LLMs (e.g., GPT-4) demonstrate superior performance in reasoning-intensive tasks like medical question answering.
• Open-source LLMs require fine-tuning to achieve competitive performance, and issues like information omission and hallucinations were observed.

Conclusions:

• Fine-tuning remains a robust strategy for BioNLP, often surpassing basic LLM prompting.
• Specific LLMs show promise for complex reasoning tasks, but require careful validation.
• Practical guidelines are needed to address LLM limitations and optimize their use in biomedical knowledge processing.