Comparing supervised machine learning and large language models in title-abstract screening

Systematic reviews require reviewers to decide on the eligibility of large numbers of articles derived from database searches. To accelerate review conduct while continuously more literature gets published, past studies proposed automating the title/abstract-screening step by either supervised machine learning or large language models. Because prior studies mainly compared results within the same model family, we directly compared common TF-IDF-based supervised baselines and a zero-shot, criteria-prompted, and open-weight large language model on the same data to discuss whether, and in which scenarios, they are feasible for review screening automation.

We predicted the eligibility of labeled articles by four supervised machine learning models (Naïve Bayes, support vector machine, random forest, logistic regression) and one large language model (Llama-3.1-8B-Instruct). Articles were labeled with eligibility as decided by human reviewers in six systematic reviews. We evaluated the performance by binary confusion matrices and calculated recall, specificity, precision, F1-score, and accuracy over a thousand bootstrap samples each. We compared these results to a reported performance of 0.86 (recall) and 0.79 (specificity) in single human reviewers.

Model performance varies greatly between the data sets. Except for Naïve Bayes, recall and specificity are closer aligned in the supervised machine learning models compared to llama. Averaged across all datasets, llama matches human recall and the Naïve Bayes classifier exceeds it, while both fall behind human specificity. Conversely, logistic regression, random forest and support vector machine fall behind human recall while all three exceed human specificity.

Both supervised machine learning and large language models achieve recalls close to or above those of human reviewers. The supervised machine learning models achieve a higher harmonic mean of recall and specificity, while the llama model is more sensitive. Considering the reliance on training data and the all-or-nothing automation with supervised machine learning, this study's results warrant their use in the extension of pre-existing, non-critical, systematic reviews. Contrarily, as large language models decide on articles individually and as they provide comprehensive, discussable, reasoning they may be used in tandem with human reviewers while the performance of ensembles of large language models is yet to be analyzed.