[Preliminary investigation of the diagnostic performance of large language models for asthma based on real-world...

Objective: To preliminarily investigate the diagnostic performance of ChatGPT-4.5 and DeepSeek-R1 large language models in assisting asthma diagnosis based on real-world clinical data. Methods: A diagnostic accuracy study design was employed. A total of 377 patients with suspected asthma who visited the respiratory outpatient department of Shanghai East Hospital between January and May 2023 and completed standardized pulmonary function tests and fractional exhaled nitric oxide (FeNO) measurements were retrospectively enrolled. Among them, there were 178 males and 199 females, with ages ranging from 16 to 84 years (45.6±13.8 years). At least two respiratory specialists independently reviewed the cases according to the 2024 GINA guidelines to establish the clinical diagnostic "gold standard." Desensitized patient demographic information, symptoms, pulmonary function data, and FeNO results were input into two models using structured prompts to obtain their diagnostic tendencies. The model diagnostic process was blinded to the clinical gold standard. Using the gold standard as reference, the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of each model were calculated. Their overall diagnostic performance was evaluated through receiver operating characteristic (ROC) curve analysis. Results: Among the 377 included samples, 167 were clinically diagnosed with asthma. The sensitivity, specificity, PPV, and NPV of ChatGPT-4.5 for diagnosing asthma were 60.48%, 95.71%, 91.82%, and 75.28%, respectively. The corresponding metrics for DeepSeek-R1 were 76.05%, 94.29%, 91.37%, and 83.19%. ROC curve analysis showed that the areas under the curve (AUC) for the two models were 0.781 and 0.852, respectively. Conclusion: The use of large language models for evaluating clinical diseases has certain diagnostic value; however, owing to the limitations of the present study design, further research is needed to provide a scientific basis for clinical application.