Comparing supervised machine learning and large language models in title-abstract screening

Area of Science:

• Bibliometrics
• Artificial Intelligence in Research

Background:

• Systematic reviews necessitate efficient article screening.
• Automating title/abstract screening using machine learning (ML) or large language models (LLMs) can accelerate review processes.
• Direct comparisons between TF-IDF-based supervised ML and zero-shot LLMs for screening automation are limited.

Purpose of the Study:

• To directly compare the feasibility and performance of common supervised ML models and a zero-shot LLM for systematic review title/abstract screening automation.
• To evaluate performance across different datasets and identify scenarios where each approach is most effective.

Main Methods:

• Four supervised ML models (Naïve Bayes, SVM, Random Forest, Logistic Regression) and one LLM (Llama-3.1-8B-Instruct) were used to predict article eligibility.
• Article eligibility was based on human reviewer decisions from six systematic reviews.
• Performance was evaluated using recall, specificity, precision, F1-score, and accuracy over 1000 bootstrap samples, compared against single human reviewer performance (0.86 recall, 0.79 specificity).

Main Results:

• Model performance varied significantly across datasets.
• Supervised ML models (except Naïve Bayes) showed aligned recall and specificity compared to the LLM.
• LLM matched human recall; Naïve Bayes exceeded it, but both fell below human specificity.
• Logistic Regression, Random Forest, and SVM had lower recall but higher specificity than humans.

Conclusions:

• Both supervised ML and LLMs achieve high recall, nearing or exceeding human levels.
• Supervised ML models offer a better harmonic mean of recall and specificity.
• LLMs are more sensitive and provide explainable reasoning, suggesting tandem use with human reviewers for critical reviews.