Performance of Zero-Shot Classifiers for Categorizing RCT Abstracts by Intervention Type: Validation Study.

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Bibliometrics

Background:

• Artificial intelligence (AI) offers potential to reduce workload in evidence synthesis and bibliometric projects.
• While instruction-tuned large language models (LLMs) are common, zero-shot classification models remain under-explored for these tasks.
• Zero-shot models are efficient, capable of categorizing text across domains with minimal hyperparameter tuning.

Purpose of the Study:

• Evaluate the performance of generalist, zero-shot classification models in categorizing interventions within rheumatology randomized clinical trials (RCTs).
• Compare zero-shot model performance against a human gold standard for intervention classification.
• Assess the utility of these models for evidence synthesis tasks.

Main Methods:

• Classified 1,054 rheumatology RCT abstracts (2009-2022) as "drug" or "non-drug" using DeBERTa and BART zero-shot models.
• Employed few-shot prompting with Llama3 8B for comparison.
• Tested variations in category labels and prompts, evaluating performance via accuracy and predictive values against human classification.

Main Results:

• Drug interventions were most common (42.9%).
• DeBERTa achieved 88.1% accuracy; Llama3 8B with few-shot prompting showed slightly higher performance.
• Model accuracy was comparable to that of an untrained human reviewer (85.9%).
• Misclassifications involved complex interventions like procedures, supplements, and biological treatments.

Conclusions:

• Zero-shot classification models demonstrate potential for simple abstract classification tasks in systematic reviews.
• These AI models achieve accuracy comparable to untrained humans, offering a tool to enhance efficiency.
• The models can supplement human reviewers, streamlining evidence synthesis and bibliometric studies.