Related Experiment Videos
Are AI chatbots ready for chikungunya public education? Evidence on validity, reliability, and readability
Qi Zhuang1,2, Jiadong Zhao1,3, Yang Liu1
1Department of Urology, Medical School, South China Hospital, Shenzhen University, Shenzhen, 518116, P. R. China.
BMC Public Health
|May 12, 2026
Summary
Newer AI chatbots show improved Chikungunya knowledge, with DeepSeek-V3.2 excelling. However, AI chatbots require careful oversight for public health use due to transparency and readability concerns.
Area of Science:
- Artificial Intelligence
- Public Health Communication
- Medical Informatics
Background:
- Chikungunya's geographic spread necessitates reliable public health guidance.
- Conversational AI is increasingly used for health information, but its accuracy and readability vary.
- Evaluating AI chatbots for medical information is crucial.
Purpose of the Study:
- To assess the medical validity, reliability, and readability of four popular AI chatbots.
- To compare the performance of different AI models and their iterations on Chikungunya-related queries.
- To determine the suitability of AI chatbots for public health information dissemination.
Main Methods:
- Four chatbots (ChatGPT, Claude, DeepSeek, Gemini) were evaluated.
- Tasks included a 50-item multiple-choice question (MCQ) dataset for validity.
- Reliability and readability were assessed using established benchmarks and clinician ratings.
Main Results:
- DeepSeek-V3.2 demonstrated the highest MCQ accuracy (86.7%), outperforming other models.
- Newer AI iterations generally showed improved performance over older versions.
- Significant differences in reliability and readability were observed across models; none met the sixth-grade reading level benchmark.
Conclusions:
- While newer AI chatbots show enhanced factual accuracy, DeepSeek-V3.2 performed best on specific tasks.
- Current AI chatbots are not ready for independent public health deployment due to limited transparency and readability issues.
- AI chatbots should supplement, not replace, authoritative public health communication, with careful consideration of ethical and governance factors.
