Benchmarking large language models for pathogen-disease classification in post-acute infection syndromes.

Area of Science:

• Computational biology and bioinformatics
• Natural language processing in medicine
• Infectious disease research

Background:

• Post-Acute Infection Syndromes (PAIS) are persistent conditions following infections like SARS-CoV-2.
• Extracting pathogen-disease associations from vast biomedical literature is challenging due to data heterogeneity.
• Limited research specifically addresses PAIS within the broader scientific corpus.

Purpose of the Study:

• To evaluate the efficacy of large language models (LLMs) in extracting pathogen-disease associations for PAIS.
• To benchmark various open-source LLMs and prompting strategies for this classification task.
• To identify optimal LLM architectures and methods for PAIS literature analysis.

Main Methods:

• A binary classification task was performed on 1000 manually labeled PubMed abstracts.
• A diverse set of open-source LLMs (4B-70B parameters) were evaluated, including generalist, reasoning, and biomedical models.
• Prompting strategies including zero-shot, few-shot, and Chain of Thought (CoT) were tested.

Main Results:

• LLM performance varied significantly based on model size, architecture, and prompting technique.
• Zero-shot prompting yielded the best results, with Mistral-Small-Instruct-2409 and Llama-3.1-Nemotron-70B-Instruct achieving high balanced accuracy and macro-F1 scores.
• Reasoning models showed improved performance with few-shot and CoT prompting, unlike generalist models.

Conclusions:

• LLMs, particularly with zero-shot prompting, are effective tools for identifying pathogen-disease associations in PAIS literature.
• Model selection and prompting strategy are critical for optimizing performance in biomedical text mining.
• This approach can aid researchers in navigating complex biomedical data for PAIS research.