Concordance Between the Multidisciplinary Team and ChatGPT-4o Decisions: A Blinded, Cross-Sectional Concordance Study...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Rheumatology

Background:

• Artificial intelligence (AI) is increasingly utilized in medical diagnostics.
• Systemic autoimmune rheumatic diseases (SARDs) often require multidisciplinary team (MDT) input for diagnosis and management.
• Comparing AI-generated decisions with expert MDT consensus is crucial for evaluating AI's clinical utility.

Purpose of the Study:

• To compare the diagnostic, treatment, and monitoring decisions made by an AI (ChatGPT-4o) with those of a multidisciplinary team (MDT) in patients with SARDs.
• To assess the concordance between AI and MDT decisions across various aspects of patient care, including diagnosis, treatment, and follow-up.
• To evaluate the potential of AI as a supportive tool in managing complex SARD cases.

Main Methods:

• A cross-sectional concordance study involving adult patients (≥18 years) with confirmed SARDs and recent MDT decisions.
• AI (ChatGPT-4o) generated single-session decisions based on standardized prompts, blinded to MDT outcomes.
• Decisions were compared across six key areas: clinical diagnosis, radiological diagnosis, anti-inflammatory treatment, antifibrotic treatment, drug-free follow-up, and further investigations.
• Cohen's Kappa (κ) statistic was used to quantify agreement between AI and MDT decisions, with R software (version 4.3.2) for analysis.

Main Results:

• The study included 47 patients, predominantly female (61.70%), with a mean age of 61.74 years. Rheumatoid arthritis (RA) was the most common diagnosis (31.91%).
• Statistically significant agreement was found across all decision types.
• Moderate agreement (κ = 0.52) was observed for clinical diagnosis. Higher concordance (κ = 0.64 and κ = 0.67) was noted for immunosuppressive treatment and drug-free follow-up decisions, respectively.
• Agreement for radiological diagnosis (κ = 0.55), antifibrotic treatment (κ = 0.49), and further investigations (κ = 0.45) was also in the moderate range.

Conclusions:

• AI demonstrates moderate concordance with MDT decisions in managing SARDs with pulmonary involvement.
• AI can provide valuable insights for clinical and radiological diagnoses, treatment selection, and follow-up planning in complex SARD cases.
• AI shows potential as a supplementary tool to support MDT decision-making in rheumatology, particularly for challenging patient cases.