Pandemic-Potential Viruses are a Blind Spot for Frontier Open-Source LLMs

Area of Science:

• Artificial Intelligence in Healthcare
• Clinical Decision Support Systems
• Infectious Disease Epidemiology

Background:

• Pre-diagnostic infectious disease triage is crucial for public health and resource allocation but remains understudied.
• Classifying symptomatic cases as viral versus non-viral at first clinical contact is a critical decision point.
• Low-resource settings, like Nigeria, face unique challenges in infectious disease management due to limited data.

Purpose of the Study:

• To evaluate the performance of large language models (LLMs) in pre-diagnostic infectious disease triage.
• To assess the effectiveness of case summaries and Retrieval Augmented Generation (RAG) for improving LLM performance.
• To develop and demonstrate alignment strategies to enhance LLM capabilities for viral vs. non-viral classification.

Main Methods:

• Creation of a benchmark dataset of first-encounter infectious disease cases from healthcare clinics in Nigeria.
• Evaluation of frontier open-source LLMs against standard tabular models for viral vs. non-viral classification.
• Implementation and assessment of Group Relative Policy Optimization with triage-oriented rewards for model alignment.

Main Results:

• LLMs underperformed traditional tabular models in pre-diagnostic triage tasks.
• Case summaries and RAG provided only modest improvements, indicating limitations of simple information enrichment.
• Models aligned with Group Relative Policy Optimization and specific rewards demonstrated consistent performance improvements.

Conclusions:

• General-purpose LLMs exhibit persistent failure modes in pre-diagnostic triage settings.
• Targeted, reward-based alignment is essential for enhancing LLM performance in clinical triage applications.
• This research highlights a pathway to improve AI-driven infectious disease diagnosis in resource-limited environments.