Artificial intelligence can match domain experts in evidence extraction and critical appraisal of microbial...

Confirmed oncogenic microbes contribute significantly to cancer burden. Identifying and confirming novel microbial oncogenicity could yield strategies and tools that will reduce disease burdens. However, relevant evidence may be dispersed across a vast biomedical literature that is infeasible for humans to comprehensively synthesize. Large Language Models (LLMs) may enable scalable, expert-level systematic evidence synthesis to identify high priority microbe-cancer pairs; however, such capabilities have not yet been demonstrated.

Domain experts were recruited to create a human-validated test dataset to benchmark the performance of LLMs (Gemini 2.5 Pro, Gemini 2.5 Flash, GPT-5, and GPT-5 Nano) on 24 original research papers using Mouse Mammary Tumor Virus-Like Virus and breast cancer as a case study. We devised a structured template for evidence extraction and appraisal of papers, consisting of multiple choice, Likert-scale, multi-select, and free-text question types (77 question items across 24 papers). Agreement between (1) experts, and (2) experts and each LLM, was determined per question instance using novel scoring metrics. LLMs were assessed by comparing inter-expert and expert-LLM agreement score distributions to determine whether LLMs behaved as additional experts by either increasing or maintaining inter-expert agreement. Free-text responses were further evaluated qualitatively.

Across all question types, LLM responses aligned closely with expert assessments, with two models (GPT-5, GPT-5 Nano) achieving score distributions statistically indistinguishable from those of experts. Gemini models behaved similarly for most tasks but were significantly more lenient in applying microbial oncogenesis criteria, often over-attributing criteria fulfillment. Hallucinations were rare, although more frequent in smaller models (Gemini 2.5 Flash, GPT-5 Nano). Methodological appraisal and identification of contradictions within full-text papers were the most persistent areas of LLM vulnerability, however, the error rate could not be directly compared with experts.

Two LLMs (GPT-5, GPT-5 Nano) were indistinguishable from domain experts on structured domain research paper evaluation tasks. This evidence supports use of LLMs for automated systematic evidence synthesis. However, methodological appraisal tasks and contradiction identification in full-text papers remain weaknesses requiring further investigation, strengthening, and possibly multi-model strategies.