Large language models in spirometry interpretation: impact of guideline-based prompting

To compare the performance of ChatGPT and Gemini with pulmonologist assessment in spirometry interpretation based on European Respiratory Society and American Thoracic Society (ERS/ATS) standards and to evaluate the effect of structured guideline-based prompting on diagnostic accuracy.

This cross-sectional diagnostic study involved 500 anonymized spirometry reports. Quality control (QC) assessment and ventilatory pattern classification were based on pulmonologist consensus as the reference standard. The same cases were evaluated by ChatGPT and Gemini before and after structured guideline-based prompting derived from ERS/ATS criteria. Analyses were restricted to 362 tests deemed interpretable by all groups. Agreement was assessed with McNemar and McNemar-Bowker tests, and diagnostic performance was evaluated with class-specific and overall multiclass measures.

Both models correctly extracted all numerical spirometric parameters but overestimated test interpretability compared with specialists (all P<0.001). Among the 362 tests interpretable by all groups, ChatGPT's overall multiclass accuracy increased from 72.1% to 74.0% after prompting (κ=0.582 to 0.608), whereas Gemini's accuracy remained 69.9% (κ=0.552 in both phases). ChatGPT's sensitivity for obstructive patterns increased from 60.0% to 63.6%, whereas Gemini's decreased from 32.7% to 19.1%; sensitivity for mixed patterns remained low. Errors predominantly involved misclassification of obstructive patterns as restrictive or mixed.

Guideline-based prompting produced model-specific and class-specific effects rather than a uniform improvement in diagnostic performance. Neither model reliably differentiated all ventilatory patterns. These findings support specialist-supervised, human-in-the-loop use and warrant standardized validation before clinical or medicolegal implementation.