Large Language Models Utility for Rapid On-Site Evaluation in Interventional Pulmonology

Area of Science:

• Pulmonology
• Pathology
• Artificial Intelligence

Background:

• Rapid on-site evaluation (ROSE) is crucial for assessing biopsy adequacy during interventional procedures.
• Integrating artificial intelligence (AI) into ROSE enhances diagnostic accuracy but faces development challenges.
• Free large language models (LLMs) offer a more accessible alternative for AI-driven diagnostic support.

Purpose of the Study:

• To assess the diagnostic accuracy of ChatGPT and Gemini LLMs in evaluating cytological smears from interventional pulmonology procedures.
• To explore the feasibility of using LLMs for AI-assisted ROSE in pulmonary diagnostics.

Main Methods:

• Retrospective analysis of cytological smears from 48 interventional bronchoscopic and ultrasound-guided biopsies (2020-2025).
• Images of ROSE-prepared samples were evaluated by ChatGPT-4o, ChatGPT-5, ChatGPT-5 "thinking", and Gemini 2.5 models.
• Comparison of LLM performance against final histopathology reports and cytologist accuracy.

Main Results:

• Cytologists achieved a balanced accuracy of 0.75.
• ChatGPT-5 "thinking" demonstrated high concordance with an accuracy of 0.65 (Gwet's AC1 = 0.81).
• Gemini 2.5 achieved an accuracy of 0.59 (Gwet's AC1 = 0.76).

Conclusions:

• This is the first study to evaluate LLM-assisted ROSE in interventional pulmonology.
• LLMs show potential for integration into pulmonary division workflows, offering accessible AI diagnostic support.
• Further prospective studies are required to validate the impact of LLMs on diagnostic yield.