Comparing a Large Language Model to Human-Generated Retention Messages for a Family Healthy Weight Program

Area of Science:

• Health Communication
• Artificial Intelligence in Healthcare
• Behavioral Science

Background:

• Text messaging effectively improves attendance and retention in community health programs.
• Message development for health programs is often resource-intensive.

Purpose of the Study:

• To compare human-generated and large language model (LLM)-generated retention messages.
• To evaluate messages based on creation time, clarity, appropriateness, and behavior change principle alignment.

Main Methods:

• Mixed-methods approach using expert ratings and developer feedback.
• Experts (n=21) rated message pairs on clarity, appropriateness, and Social Cognitive Theory alignment.
• Statistical tests (Wilcoxon signed-rank, equivalence testing) compared message types.

Main Results:

• LLM messages received significantly higher clarity ratings (4/5 pairs) and appropriateness ratings (3/5 pairs).
• Messages showed statistical equivalence in behavior change alignment for 3/5 pairs.
• Development time was similar, but LLM messages were more concise.

Conclusions:

• LLMs show potential for efficient health message creation, reducing workload for professionals.
• LLMs can generate clear and appropriate retention messages for health programs.
• While LLMs can assist, human oversight remains important in message development.