Benchmarking large language models for biomedical natural language processing applications and recommendations.

Area of Science:

• Biomedical Informatics
• Natural Language Processing
• Artificial Intelligence

Background:

• The exponential growth of biomedical literature necessitates automated knowledge extraction.
• Biomedical Natural Language Processing (BioNLP) offers a solution for manual curation challenges.
• The efficacy of Large Language Models (LLMs) in specialized BioNLP tasks requires systematic evaluation.

Purpose of the Study:

• To systematically evaluate the performance of leading LLMs (GPT, LLaMA) on diverse BioNLP tasks.
• To compare LLM performance (zero-shot, few-shot, fine-tuning) against established models (BERT, BART).
• To identify practical challenges and provide insights for LLM application in BioNLP.

Main Methods:

• Evaluation of four LLMs across 12 BioNLP benchmarks and six application types.
• Comparative analysis of zero-shot, few-shot, and fine-tuning strategies for LLMs.
• Assessment of LLM outputs for inconsistencies, missing information, and hallucinations, alongside cost analysis.

Main Results:

• Traditional fine-tuning of models like BERT/BART generally surpasses zero- or few-shot LLM performance on most BioNLP tasks.
• Closed-source LLMs, such as GPT-4, demonstrate superior capabilities in reasoning-intensive applications like medical question answering.
• Open-source LLMs require fine-tuning to bridge performance gaps, and all evaluated LLMs exhibited issues with factual accuracy and completeness.

Conclusions:

• LLMs present a promising avenue for BioNLP, but their practical application requires careful consideration of model choice and fine-tuning strategies.
• Fine-tuning remains crucial for optimizing LLM performance in BioNLP, particularly for open-source variants.
• Addressing challenges like hallucinations and missing information is essential for reliable LLM deployment in biomedical knowledge synthesis.