A biomedical BERT ensemble (TITAN-SR) outperformed active learning and LLM chatbots for systematic-review title and...

Machine learning tools for literature screening have existed for a decade, yet adoption remains limited: active learning needs hundreds of decisions before prioritizing well, and commercial large language model (LLM) chatbots are used without task-specific training. We aimed to develop TITAN-SR (Training Infrastructure for Automated Nomination in Systematic Reviews), a screening tool built on biomedical Bidirectional Encoder Representations from Transformers (BERT) models, and to compare it with the active-learning tool ASReview and commercial LLM chatbots.

We assembled 762,934 citation-label pairs from 19,787 completed reviews and trained TITAN-SR on a prespecified review-level split, testing on held-out reviews. The tool combines biomedical BERT models (PubMedBERT and BioLinkBERT) that read a review's eligibility criteria with each citation's title and abstract; training penalized false negatives 10 times more heavily than false positives. We compared TITAN-SR with four chatbots used zero-shot on a stratified 500-review benchmark (Claude Sonnet 4, GPT-4o, DeepSeek-V3, Gemini 2.0 Flash), with GPT-4o-mini on all 3,434 evaluable test reviews, and with ASReview on 22 temporally independent Cochrane reviews (142,504 records); ASReview received a warm-up of 20% of known includes.

Across 3,434 test reviews, TITAN-SR achieved a median specificity at 99% recall of 0.902 and median area under the receiver operating characteristic curve (AUC) of 0.974. It outperformed all four chatbots (Bonferroni-corrected paired Wilcoxon p ≤ 5 × 10^-31) and GPT-4o-mini. The chatbots showed calibration collapse: 82-99% of confidence scores were ≥ 0.9 regardless of accuracy. On the 22 external reviews performance was virtually identical (median AUC 0.976; specificity at 95% recall 0.897), and at a threshold fixed on the internal validation split TITAN-SR met the pre-registered primary criterion, retaining 99.8% of included studies (95% confidence interval [CI] 99.5 to 100.0%). It outperformed ASReview on 18 of 22 reviews (median paired difference in specificity at 95% recall +0.145, 95% CI +0.101 to +0.201).

A purpose-built screening tool based on biomedical BERT models, developed from nearly 20,000 completed reviews, outperformed ASReview and commercial LLM chatbots evaluated without task-specific training. Temporal validation on 22 held-out Cochrane reviews confirmed generalization to reviews published after the model-development period.