Scientific evidence of commercial artificial intelligence products for pulmonary nodule assessment on CT scans: a systematic review

This systematic review evaluates the available evidence on the efficacy of commercially available AI-software applications for lung nodule assessment on CT scans.

In adherence to PRISMA guidelines, a thorough search of electronic databases was conducted (January 2012-November 2024) to identify studies on CE-marked and/or FDA-cleared AI-based systems for evaluating pulmonary nodules on CT scans. Articles were systematically categorised using the Radiology AI Deployment and Assessment Rubric (RADAR) to assess the evolution of efficacy over time across various hierarchical levels.

A total of 95 studies were included. Between 2012 and 2024, the number of studies increased from 14 in 2012-2017 to a total of 95 by 2024. The studies were categorised using the RADAR efficacy model. In the early period (2012-2017), most studies focused on lower efficacy levels, with Level-1 (technical efficacy) and Level-2 (diagnostic accuracy) dominating at 83.3%. During this period, AI applications were mainly focused on quantification (46.7%), while malignancy prediction was addressed in only 6.7% of studies. By 2024, Levels-3 (diagnostic thinking efficacy), 4 (therapeutic efficacy), and 5 (patient outcomes) accounted for over a third of all studies. Malignancy prediction further increased to 28.6%, and nodule characterisation emerged in 4.5% of studies. Despite advancements, research on patient outcomes and cost-effectiveness efficacy (Levels-5 and 6) remains limited. Also, all included studies demonstrated high risk of bias in at least one domain, and nearly two-thirds involved vendor funding or co-authorship.

In conclusion, the growing interest and investment in AI technologies for thoracic radiology have driven significant advancements in lung nodule assessment on CT-scans. Gaps remain in assessing patient outcomes and societal implications, which must be addressed to fully realise AI's potential in clinical practice and public health.

Question What is the current level of scientific evidence supporting commercially available AI tools for pulmonary nodule assessment on CT, and is this sufficient to support their clinical implementation? Findings Research on AI for pulmonary nodule assessment has surged, with the focus evolving from basic technical performance in the early years to higher clinical outcomes by 2024. Clinical relevance This study provides critical insights into the evolving role of AI in lung nodule assessment on CT-scans, highlighting advancements in technology and identifying key gaps in current research. Addressing these is essential for optimising AI's clinical application and improving patient outcomes.