Training an AI Chatbot to Manage Health in Underserved Populations: Methodological Approach

Health disparities such as morbidity and mortality among childbearing women remain high in the United States, especially among those with risks associated with criminal legal system involvement. These underserved women are often managed through community supervision such as probation. They have many needs and could benefit from easily accessible mobile health (mHealth) apps that specifically target their health and safety using artificial intelligence (AI).

The purpose of this methodological case study is to provide our detailed strategies and findings for systematically designing, optimizing, and testing an AI chatbot.

This methodological case study used an mHealth app's AI chatbot, JUN, that involved preliminary studies and development efforts to support childbearing women on community supervision. We applied the Information Systems Research framework to guide the steps on how we designed, tailored, configured, and tested the chatbot using a retrieval-augmented generation framework. We demonstrated the feasibility of using an in-context learning approach addressing relevance, design, and rigor cycles.

During both crisis and noncrisis situations, the JUN chatbot had an overall performance of 89% accuracy (N=178) in detecting a "crisis." Qualitative findings displayed increased usability of JUN to manage health at night by participants. The findings also demonstrated that the role of caregiving or current pregnancy was a motivating factor to manage health using technology such as the JUN app. Collectively, the sample expressed that barriers to managing their health effectively were associated with limited transportation, time off work, and insurance coverage. Participants in the community supervision group also described that stress related to criminal legal system involvement put limitations in how they managed their health and well-being. Altogether, participants from both groups discussed how an anonymous chat feature and app store accessibility would enhance the usability and acceptability of JUN among users. Pregnant women used the app to manage feelings of fatigue, shortness of breath, food cravings, anxiety, confidence, determination, frustration, excitement, happiness, hopefulness, irritation, love, as well as acknowledgment of their own feelings. Pregnant participants on community supervision had more housing (P=.05) and food (P=.01) insecurity, worry about electricity being turned off (P=.04), and needing resources (P=.01) compared to pregnant women without community supervision.

We illustrate the methodological case study to design, optimize, and test an AI chatbot within an mHealth app to provide health and safety-related support for childbearing women on community supervision. This methodological case study poses possibilities for further development and testing of interventions for populations with similar risks to their health and safety.

ClinicalTrials.gov NCT06636110; https://clinicaltrials.gov/ct2/show/NCT06636110.