Methodological insights into ChatGPT's screening performance in systematic reviews.

Area of Science:

• Radiology
• Artificial Intelligence
• Medical Informatics

Background:

• Systematic review screening is time-intensive.
• Machine learning requires training data and annotation.
• Large language models (LLMs) like ChatGPT offer potential for automated screening.

Purpose of the Study:

• To evaluate ChatGPT's efficacy in automating radiology systematic review abstract screening.
• To assess ChatGPT's performance without prior training data.
• To compare ChatGPT's screening accuracy and efficiency against general physicians.

Main Methods:

• Prospective simulation study comparing ChatGPT and general physicians (GPs).
• Evaluated 1198 radiology abstracts across three subfields.
• Assessed sensitivity, specificity, PPV, NPV, workload savings, and inter-rater agreement (Kappa).

Main Results:

• ChatGPT screened abstracts in under an hour; GPs took 7-10 days.
• ChatGPT achieved 95% sensitivity and 99% NPV, outperforming GP consensus.
• Workload savings ranged from 40-83%, but ChatGPT had lower specificity and PPV (Kappa=0.27).

Conclusions:

• ChatGPT demonstrates significant potential for automating systematic review screening in radiology.
• It offers high sensitivity and substantial workload reduction.
• Serves as an efficient first-line tool, complementing human expertise.