Evaluating AI chatbots for providing health advice on occupational sedentary behavior in the workplace

Office workers accumulate most of their waking h in sedentary behavior, yet scalable personalized occupational-health advice remains limited, and the quality of large language model (LLM)-based tools in this domain has not been rigorously examined.

This study evaluated whether a domain-specialized chatbot system for providing health advice on sedentary behavior, integrating retrieval augmentation, a curated corpus, and safety guardrails, performs better than a general-purpose LLM and static educational materials.

A sequential explanatory mixed-methods design was used. In the blinded expert evaluation, nine raters independently rated 60 responses across five quality dimensions. In the user survey, 654 eligible Chinese office workers interacted with the specialized chatbot and completed six perception scales (two adapted from established technology-acceptance frameworks and four developed for this study) plus open-ended feedback.

Inter-rater reliability was good for the composite score; in a case-level safety screen, no response from the specialized chatbot or the standard materials was flagged as potentially unsafe, whereas two Doubao responses were flagged by two or more raters. The specialized chatbot achieved the highest composite rating, followed by Doubao and standard materials. In a linear mixed-effects model and crossed random intercepts for rater and question, all Holm-adjusted pairwise contrasts were significant. Advantages of the specialized chatbot over Doubao were most pronounced on safety in symptom-management and special-population items. Among users, perceived ease of use was highest, whereas perceived safety and intention to use were rated comparatively lower. Frequent prior AI-chatbot use was associated with higher ratings on all six scales; after covariate adjustment the association persisted for five of the six scales but was attenuated and no longer significant for perceived safety.

The specialized chatbot system received expert-rated accuracy and safety scores that did not differ significantly from standard educational materials, but exceeded standard materials on usefulness, comprehensibility, and completeness, and outperformed the general-purpose LLM on accuracy, safety, completeness, and the overall composite score. These findings concern response quality and immediate user perceptions, not behavioral or clinical outcomes. Nonetheless, users remained cautious about safety and intention to use after this brief exposure.