A biomedical BERT ensemble (TITAN-SR) outperformed active-learning and large language model chatbots for systematic...

Machine-learning tools for literature screening have existed for a decade, yet adoption remains limited: active-learning needs hundreds of decisions before prioritizing well, and commercial large language model (LLM) chatbots are used without task-specific training. We aimed to develop TITAN-SR (Training Infrastructure for Automated Nomination in Systematic Reviews), a screening tool built on biomedical Bidirectional Encoder Representations from Transformers (BERT) models, and to compare it with the active-learning tool ASReview and commercial LLM chatbots.

We assembled 762,934 citation-label pairs from 19,787 completed reviews and trained TITAN-SR on a prespecified review-level split, testing on held-out reviews. The tool combines biomedical BERT models (PubMedBERT and BioLinkBERT) that read a review's eligibility criteria with each citation's title and abstract; training penalized false negatives 10 times more heavily than false positives. We compared TITAN-SR with four chatbots used zero-shot on a stratified 500 review benchmark (Claude Sonnet 4, GPT-4o, DeepSeek-V3, Gemini 2.0 Flash), with GPT-4o-mini on all 3434 evaluable test reviews, and with ASReview on 22 temporally independent Cochrane reviews (142,504 records); ASReview received a warm-up of 20% of known includes.

Across 3434 test reviews, TITAN-SR achieved a median specificity at 99% recall of 0.902 and median area under the receiver operating characteristic curve (AUC-ROC) of 0.974. It outperformed all four chatbots (Bonferroni-corrected paired Wilcoxon P ≤ 5 × 10^-31) and GPT-4o-mini. The chatbots showed calibration collapse: 82%-99% of confidence scores were ≥0.9 regardless of accuracy. On the 22 external reviews, performance was virtually identical (median AUC 0.976; specificity at 95% recall 0.897), and at a threshold fixed on the internal validation split, TITAN-SR met the preregistered primary criterion, retaining 99.8% of included studies (95% confidence interval [CI] 99.5% to 100.0%). It outperformed ASReview on 18 of 22 reviews (median paired difference in specificity at 95% recall +0.145, 95% CI +0.101 to +0.201).

A purpose-built screening tool based on biomedical BERT models, developed from nearly 20,000 completed reviews, outperformed ASReview and commercial LLM chatbots evaluated without task-specific training. Temporal validation on 22 held-out Cochrane reviews confirmed generalization to reviews published after the model development period.

Screening thousands of search results is one of the most time-consuming steps in a systematic review. We built Training Infrastructure for Automated Nomination in Systematic Review (TITAN-SR), a low-cost tool that runs on an ordinary computer and reads each article's title and summary together with the review's eligibility criteria to flag the articles most likely to be relevant. Across thousands of completed reviews, it kept about 99% of the relevant articles, while setting aside about 90% of the irrelevant ones, and in our evaluation, it identified relevant studies more effectively than two widely used alternatives (an active-learning tool and general purpose artificial intelligence chatbots). It is intended to help reviewers prioritize their reading and act as a second screener, not to replace human judgment; the model is available to researchers on request.