Accuracy and reproducibility of a large language model in assessing spirometry test acceptability and interpretation: a performance study

Spirometry is one of the most used pulmonary function tests for assessing respiratory function and diagnosing ventilatory disorders in clinical practice. Evaluation of test validity and interpretation is typically performed by trained healthcare professionals. The present study aims to evaluate the capability of a publicly available large language model (LLM) to perform quality assurance and interpretations of spirometry tests.

A total of 150 spirometry tests were retrospectively collected from the pulmonary function laboratories. The spirometry tests were evaluated for acceptability and interpreted by clinicians according to ATS/ERS recommendations as part of standard clinical practice. Subsequently, a publicly available LLM (ChatGPT 5.5) was prompted to evaluate the acceptability and interpret the same spirometry tests according to ATS/ERS criteria. The model interpreted each spirometry test on three separate occasions for assessing reproducibility. Accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV), with 95% confidence intervals (CIs), were calculated by comparing LLM assessments with clinician assessments of spirometry test acceptability and interpretation.

For spirometry acceptability evaluation, ChatGPT demonstrated an overall accuracy of 87% (95% CI: 81%-92%), sensitivity of 91% (95% CI: 85%-95%), specificity of 61% (95% CI 39%-80%), PPV of 94% (95% CI: 89%-97%), and NPV of 48(95% CI: 29%-67%) with Cohen's k of 0.46. The spiromteric interpretation agreement with the clinicians reference standard was moderate, with exact agreement of 0.61 (95% CI: 0.54-0.69) while Cohen's Kappa (Unweighted) was 0.47 (95% CI: 0.37-0.56) and Macro F1 score was 0.39 (95% CI: 0.33-0.49). Reproducibility across three independent trials was moderate (Fleiss' κ 0.58). Class-specific interpretation performance varied by category; sensitivity was highest for normal pattern and restrictive defects (0.72), whereas sensitivity for obstructive defects was 0.27. F1-scores were highest for normal and restrictive defects (0.81 and 0.62, respectively).

The current findings suggest that ChatGPT 5.5 has a limited capability, at its current stage of development, to serve as a reliable tool for independent spirometry interpretation in clinical practice. Nevertheless, it may serve as an adjunct to clinical decision-making and educational tools. Prospective controlled studies are needed to confirm these findings.