Accuracy, Usefulness, and Impact Variability of ChatGPT-4 for COPD Medication Management: A Modified Delphi Study

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Pharmacy Practice
• Respiratory Medicine

Background:

• Chronic obstructive pulmonary disease (COPD) management presents complex challenges.
• Large language models (LLMs) like ChatGPT show potential for generating treatment plans.
• The reliability of ChatGPT for COPD care is not well-established.

Purpose of the Study:

• To assess the accuracy, usefulness, and consistency of ChatGPT-4.0 responses for COPD medication management.
• To evaluate the variability in clinical impact of simultaneous AI-generated treatment recommendations.

Main Methods:

• Five distinct COPD treatment scenarios were posed to ChatGPT-4.0 on three separate computers.
• Fifteen responses were generated and evaluated by three clinical pharmacists using a modified Delphi process.
• Ratings for accuracy, usefulness, and impact variability were collected on a 0-2 scale.

Main Results:

• Consensus was reached on 88.9% of the 45 evaluated response-domain ratings.
• Accuracy varied from poor to good, usefulness from somewhat to very useful, and impact variability from low to high.
• Inconsistent responses were observed, including outdated guideline references and varied usefulness for complex cases.

Conclusions:

• Simultaneous ChatGPT-4.0 outputs for identical COPD prompts demonstrated material differences in accuracy and potential clinical impact.
• AI-generated recommendations require careful review and should supplement, not replace, clinical judgment.
• Deployment of LLMs in COPD management necessitates adherence to current guidelines and local oversight.