Evaluating AI-Mediated Health Communication via Large Language Model-Based Frequently Asked Questions Rewriting to...

Effective communication about clinical trials is essential, as low enrollment undermines scientific validity and contributes to health care inequities. However, recruitment remains a persistent challenge, particularly among older adults, minority populations, and individuals with limited health literacy. Although large language models (LLMs) show promise in understanding and generating health information, it is unclear whether these generative artificial intelligence (AI) tools can improve the content of hospitals' frequently asked questions (FAQ) pages to enhance public attitudes and intentions toward clinical trial participation.

This study aimed to compare clinical trial FAQ from health organizations and hospitals with versions rewritten by LLMs to examine whether the generated content improves public attitudes and intentions toward clinical trial participation and to identify the mechanisms underlying these effects.

A total of 308 question-answer pairs were collected from the FAQ pages of 38 health organizations and hospitals, categorizing them into 52 types and selecting the 11 most frequent for testing. A comparative survey experiment was conducted with 440 participants randomly assigned to one of the two survey stimuli: the original FAQ versus the GPT-4o-generated answers emphasizing comprehension and empathy. The study compared the impact of AI-generated versus standard FAQ content on attitudes toward clinical trials and examined Theory of Planned Behavior constructs to determine for whom and how AI information is most effective.

Participants were recruited through CloudResearch, yielding a 96.94% completion rate, resulting in 440 valid responses across the 2 types of content exposure. Participants who viewed GPT-4o-generated information (mean 0.26, SD 0.65) showed a marginally greater positive change in outcome evaluation attitudes than those who viewed standard FAQ (mean 0.13, SD 0.70; P=.05; 95% CI 0.00-0.25). Follow-up linear regression analyses revealed that several individual factors significantly moderated the effect of the information type (FAQ vs GPT-4o) on attitude change, including age (mean difference 0.87, SE 0.33; 2-tailed t394=2.62; P=.009); race (mean difference 0.36, SE 0.15; t383=2.47; P=.01); risk aversion (B=0.12; SE 0.06; t383=2.23; P=.03); fear of ineffective treatment (B=0.11; SE 0.05; t383=2.03; P=.04); and fear of unknown treatment effects (B=0.21; SE 0.07; t383=3.10; P=.002).

This study is the first to apply the Theory of Planned Behavior to compare LLM-rewritten versus original FAQ content for clinical trial communication. The findings show that the GPT-4o-generated responses improved attitudes among traditionally underrepresented groups, including older adults, Black participants, and those with higher uncertainty avoidance or treatment concerns. These attitude gains were positively linked to participation intentions, suggesting that AI-generated language can enhance public attitudes, perceptions, and engagement with clinical research.