Comparing Large Language Model AI and Human-Generated Coaching Messages for Behavioral Weight Loss

Area of Science:

• Behavioral Science
• Artificial Intelligence
• Health Coaching

Background:

• Automated weight control coaching messages can be repetitive and less effective than human coaching.
• Large language model (LLM) AI chatbots offer potential for personalized and novel coaching messages.
• The feasibility and acceptability of LLM-based weight management coaching remain understudied.

Purpose of the Study:

• To assess the feasibility and acceptability of LLM AI (ChatGPT) for generating weight control coaching messages.
• To compare the helpfulness of human-written versus AI-generated coaching messages.
• To explore user perceptions of AI-generated messages in a weight-loss trial.

Main Methods:

• Eighty-seven adults with a BMI ≥ 27 kg/m² participated in a weight-loss trial.
• Participants rated the helpfulness of human-written and ChatGPT-generated coaching messages on a 5-point Likert scale.
• Qualitative feedback and AI message identification were collected in two phases with message revisions.

Main Results:

• Initial AI messages were rated less helpful, but revised AI messages matched human messages in helpfulness (82% rated ≥3).
• 50% of AI messages were misidentified as human-written, indicating sophisticated mimicry.
• Participants found AI messages empathetic and personalized but sometimes formulaic and data-focused.

Conclusions:

• LLM AI shows preliminary feasibility for creating effective weight control coaching messages.
• AI-generated messages can be perceived as helpful and personalized, rivaling human-written content after refinement.
• Further enhancements are needed to address perceived formulaic and data-centric aspects of AI coaching.