Psychological support for depression and anxiety across chronic illness contexts: a comparative evaluation of three general-purpose AI chatbots

Digital mental health tools, including general-purpose artificial intelligence (AI) chatbots, may provide accessible and low-threshold psychological support for people living with chronic conditions. However, their value in this population depends not only on recognizing depression or anxiety, but also on understanding how psychological distress is shaped by disease-specific experiences and when self-guided support should transition toward professional care. We evaluated the appropriateness, contextual sensitivity, safety, and consistency of psychological support provided by three widely accessible AI chatbots across contrasting chronic illness contexts.

We conducted a standardized simulated-patient benchmarking study comparing ChatGPT, Gemini, and Doubao. Twenty scenarios were developed across type 2 diabetes mellitus (T2DM) and chronic spontaneous urticaria (CSU), representing matched mild and moderate presentations of depression and anxiety together with condition-specific distress related to diabetes self-management and CSU-related flare uncertainty. Each scenario was independently submitted to all three systems in two testing rounds, yielding 120 responses. Blinded raters assessed overall psychological support appropriateness, recognition and emotional support, chronic-illness contextualization, psychological support and referral, psychosomatic balance and safety, and major clinical or safety errors. Repeated testing assessed response consistency.

Overall psychological support appropriateness differed across systems (P = 0.002), with 90.0% of ChatGPT, 75.0% of Gemini, and 55.0% of Doubao responses meeting the prespecified appropriateness threshold. Recognition and emotional support was consistently strong across systems, whereas greater differences emerged in chronic-illness contextualization (P = 0.001), psychological support and referral (P = 0.002), and psychosomatic balance and safety (P<0.001). All responses to mild presentations were classified as psychologically appropriate, compared with 58.3% of responses to moderate presentations, indicating that performance became less reliable when digital support required stronger escalation toward professional assessment. Contextualization was also less consistent in CSU, particularly when responses needed to incorporate flare unpredictability, pruritus, sleep disruption, and illness uncertainty. Major clinical or safety errors occurred in 0%, 10.0%, and 15.0% of first-round ChatGPT, Gemini, and Doubao responses, respectively. Test-retest consistency was moderate and varied between systems.

General-purpose AI chatbots show potential as accessible, self-guided digital mental health resources for people living with chronic conditions, particularly for recognizing distress and providing initial supportive guidance. Their current limitations emerge when effective support requires disease-specific contextualization, balanced interpretation of psychological and somatic factors, or escalation as symptoms become more severe. These findings identify important boundaries for the use of conversational AI in chronic-illness mental health care and suggest that implementation should emphasize context-sensitive guidance, severity-responsive referral, and clear integration with rather than substitution for professional care.