Related Experiment Video
Updated: Oct 24, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Evaluating Chinese large language models for HIV/AIDS health information: a multidimensional comparative study of
Jinhao Gu1, Jianwei Liu1, Yunrong Liu1
1Department of Biomedical Informatics, School of Life Sciences, Central South University, Changsha, Hunan, China.
Background:
Chinese large language models (LLMs) are increasingly used for HIV/AIDS health information, yet their multidimensional performance remains unevaluated. This study systematically assessed five mainstream Chinese LLMs across accuracy, caring, completeness, understandability, and actionability.
Methods:
A cross-sectional evaluation was conducted using 50 standardized HIV/AIDS questions. Each model's responses were rated by three independent experts on five dimensions, including adapted PEMAT-based items. Latent profile analysis (LPA) and partial correlation network analysis were employed to identify quality typologies and interdimensional dependencies. Chi-square tests examined associations between profile membership and model identity or question domain.
Results:
The four general-purpose LLMs showed high accuracy (mean > 4.5/5) and completeness (86%-98% full marks), with no significant differences among them (all P-FDR > 0.05). HuatuoGPT-II, the only medical-domain-specific model, scored significantly lower on both dimensions (all P < 0.001). Its responses were less consistent with current guidelines and included one instance of implicit stigmatizing language. All models showed systemic deficits in caring (median = 3.0/5) and actionability (max = 1.70/3). Understandability showed a ceiling effect and no significant profile-level differences, although small model-level differences were observed. LPA identified four profiles: High Accuracy and Completeness (50.0%), Moderate-Balanced (3.6%), Low-Quality (18.8%), and High Completeness and Actionability-Oriented (27.6%). Network analysis identified completeness as the most central node (strength = 0.813) and actionability as a key bridge (betweenness = 0.167). A weak negative partial correlation between accuracy and actionability (r = -0.083) suggested a potential trade-off between technical precision and practical guidance at the statistical level. Profile distributions differed by model (χ2 = 124.99, P < 0.001) but not domain (P = 0.625), indicating heterogeneity stems from model-specific characteristics. Yuanbao-Hy3 showed the most balanced humanistic-technical profile. DeepSeek-V4 excelled technically but lacked caring.
Conclusions:
In this evaluation, the four general-purpose models outperformed the single medical-domain-specific model examined in factual accuracy and completeness. However, systemic deficits in empathetic communication and actionable guidance persist. Model selection should prioritize holistic profile distributions over single-dimension scores. Developers must integrate de-stigmatizing corpora and action-oriented alignment; clinicians should position AI as supplementary to professional care. Future evaluations should adopt person-centred, multi-turn, and real-user validated frameworks.