Accuracy of ChatGPT-4o in Text and Video Analysis of Laryngeal Malignant and Premalignant Diseases

Chatbot Generative Pretrained Transformer (ChatGPT), a multimodal generative AI, has been studied for potential applications in healthcare, including otolaryngology-head and neck surgery. In this study, authors investigates the consistency of ChatGPT-4o in analyzing clinical fiberoptic videos of suspected laryngeal malignancies compared to expert clinicians.

This experimental study involved twenty patients with primary laryngeal disease consulting at a tertiary academic center. Data, including laryngeal fiberoptic video examinations, were retrospectively analyzed using the ChatGPT-4o application programming interface. Responses were assessed for diagnostic accuracy, consistency, and clinical recommendations. Three otolaryngology-head and neck consultants independently evaluated ChatGPT-4o's performance using the Artificial Intelligence Performance Instrument and a five-point Likert scale for complexity and consistency.

ChatGPT-4o identified malignant diagnoses as the primary diagnosis in 30% of cases, while proposing malignancies as one of the top three diagnoses in 90% of cases. Despite high sensitivity, specificity was limited. The mean consistency score for image analysis was 2.36 ± 1.13, with an intraclass correlation coefficient of 0.890 (P = 0.03). The model showed a tendency to prioritize text over visual data, limiting the improvement in diagnostic accuracy from video input.

While ChatGPT-4o demonstrates potential in analyzing laryngeal pathologies through multimodal data, current limitations in specificity and image interpretation indicate the need for further refinement. Ongoing advancements could enhance its integration into clinical workflows, supporting accurate diagnoses and decision-making in otolaryngology.