Related Experiment Video
Updated: May 18, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
A bilingual evaluation of chatbot performance in bruxism-related information: accuracy and readability across two
Merve Cennet Altuntaş1, Gülbahar Erdinç Akyol2
1Department of Prosthodontics, Faculty of Dentistry, Bilecik Şeyh Edebali University, Bilecik, Türkiye.
Background:
Artificial intelligence (AI) chatbots have increasing applications in healthcare; however, their accuracy and readability across different languages remain unclear. Therefore, this study aimed to compare the performance of ChatGPT-5 and DeepSeek-V3 on bruxism-related questions in English and Turkish.
Methods:
Responses generated by ChatGPT-5 and DeepSeek-V3 to 20 questions were evaluated in Turkish and English, yielding four chatbot-language conditions. Accuracy was independently scored by two prosthodontists using a Modified Global Quality Score, and inter-rater reliability was assessed using the intraclass correlation coefficient (ICC). Readability was defined as the ease of reading and was assessed using the Flesch Reading Ease (FRE) scale for English texts and the Ateşman readability formula for Turkish texts. Descriptive statistics were calculated, differences in accuracy across conditions were analyzed using the Friedman test, and the relationship between accuracy and readability was examined using Spearman's rank correlation coefficient.
Results:
Accuracy scores of responses generated by ChatGPT-5 and DeepSeek-V3 were generally high across both Turkish and English. No statistically significant differences in accuracy were observed among the four model-language combinations based on the Friedman test (χ² (3) = 3.204, p=0.361). Inter-rater reliability analysis demonstrated moderate agreement for single measures and good agreement for average measures between the two evaluators (ICC = 0.623 and 0.767, respectively). Comparison of Atesman readability scores and FRE-based English readability analyses revealed no statistically significant differences between the two chatbots, as confirmed by Mann-Whitney U tests (Turkish: p = 0.646; English: p = 0.745).
Conclusion:
ChatGPT-5 and DeepSeek-V3 demonstrated comparable accuracy in bruxism-related responses across Turkish and English. However, high readability levels limit patient accessibility, and no association was found between accuracy and readability, indicating that simplification is necessary for effective patient education.
