Related Experiment Video
Updated: Jul 12, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Information quality, readability, and empathy of AI-generated public mental health information: a comparative
Qilong Wang1,2, Siheng Ma3, Jian Sun1
1Xihua University, Chengdu, Sichuan, China.
Background:
Mental disorders are a growing global health burden, yet healthcare resources remain scarce. Large language models (LLMs) may support public mental health information seeking, but their information quality, readability, and empathy in psychiatric contexts require validation.
Method:
We developed a test bank of 48 public mental health questions from literature, Google Trends (2004-2025), and clinical consultations. Eight LLM chatbots were compared for information quality, source transparency, readability, and empathy using established rating instruments, readability indices, and psychiatrist-rated and user-perspective empathy assessments. Statistical analysis used Kruskal-Wallis and Dunn's post-hoc tests, with Spearman correlations.
Results:
Gemini 3.0 Pro and GPT-5.2 Think showed relatively higher information quality and source transparency scores, whereas spontaneous source transparency was poor across models, with median JAMA scores of 0. No model met the sixth-grade readability standard; Claude Sonnet 4.5 generated relatively more readable responses, whereas Claude Sonnet 4.5 Think produced responses with the highest reading difficulty. In the psychiatrist-rated empathy assessment, Gemini 3.0 Pro and DeepSeek-V3 showed the highest high-empathy response rates, at 54.2% and 43.8%, respectively. User-perspective empathy ratings were generally lower, with DeepSeek-V3 and Gemini 3.0 Pro showing the highest user-perspective high-empathy response rates, at 31.2% and 27.1%, respectively. Information quality and source transparency metrics showed only weak correlations with readability metrics.
Conclusion:
LLMs face important challenges in psychiatric information delivery. Gemini 3.0 Pro and GPT-5.2 Think showed higher information quality and source transparency scores and better information structuring, whereas Claude Sonnet 4.5 generated relatively more readable responses. However, source opacity and high reading difficulty limit direct patient-facing use. Empathy varied across models and differed between psychiatrist-rated and user-perspective assessments, suggesting that empathic communication requires separate optimization and validation with intended users. Future work should balance information quality, source transparency, readability, traceability, safety, and emotionally appropriate responses.
