ChatGPT Health performance in a structured test of triage recommendations

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Decision Support

Background:

• OpenAI launched ChatGPT Health in January 2026 as a consumer health tool.
• AI-powered triage systems are increasingly being developed for public use.

Purpose of the Study:

• To conduct a structured stress test of ChatGPT Health's triage recommendations.
• To identify potential safety concerns and failure modes in AI-driven medical triage.

Main Methods:

• A stress test was performed using 60 clinician-authored vignettes across 21 clinical domains.
• A total of 960 responses were analyzed under 16 factorial conditions.
• The impact of anchoring bias and crisis intervention message activation was evaluated.

Main Results:

• Performance showed an inverted U-shaped pattern, with failures at clinical extremes (non-urgent and emergency).
• The system under-triaged 52% of gold-standard emergencies, including diabetic ketoacidosis and impending respiratory failure.
• Anchoring bias significantly shifted triage recommendations toward less urgent care (OR 11.7).
• Crisis intervention messages for suicidal ideation activated unpredictably.

Conclusions:

• ChatGPT Health demonstrated significant safety concerns, including missed high-risk emergencies and inconsistent crisis intervention.
• Prospective validation is crucial before the consumer-scale deployment of AI triage systems.
• AI triage systems require rigorous testing to ensure patient safety and efficacy.