The health risks of generative AI-based wellness apps

Area of Science:

• Digital Health
• Mental Healthcare Technology
• Regulatory Science

Background:

• Artificial intelligence (AI)-enabled chatbots are increasingly integrated into mental wellness applications.
• Current 'wellness' applications, particularly generative AI-powered ones, operate within a regulatory gray area, often avoiding health regulator review.
• Users are increasingly utilizing these apps for sensitive mental health disclosures and crisis support.

Purpose of the Study:

• To examine the regulatory landscape surrounding AI-enabled wellness apps.
• To identify potential health risks associated with the use of these applications.
• To discuss the implications for app developers and regulatory bodies.

Main Methods:

• Perspective piece analyzing the current state of AI wellness apps and their regulation.
• Discussion of user interaction patterns and potential adverse outcomes.
• Review of existing US regulatory frameworks and their limitations.

Main Results:

• AI wellness apps present risks as users share mental health problems and seek crisis support.
• App responses can sometimes exacerbate user harm, a challenge not adequately addressed by current US regulations.
• The line between wellness apps and medical territory is blurring, creating regulatory challenges.

Conclusions:

• There is a critical need to address the regulatory gaps for AI-enabled mental wellness applications.
• Ensuring user safety requires a re-evaluation of how these technologies are regulated, especially when they approach medical territory.
• Developing clear guidelines and regulatory priorities is essential for responsible innovation in AI mental health tools.