Do Large Language Models Perform Equally Across Languages? A Comparison of Responses to Frequently Asked Questions in...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Anesthesiology

Background:

• Large language model (LLM) chatbots are increasingly used in healthcare for patient education.
• Evaluating the reliability and understandability of LLM responses in multiple languages is crucial, especially in anesthesia.
• Patient education in anesthesia requires accurate and accessible information.

Purpose of the Study:

• To compare the quality of English responses from ChatGPT 4.0 and DeepSeek V3.
• To evaluate content and communication differences between English and Turkish responses from these LLMs.
• To assess the suitability of LLMs for anesthesia patient education.

Main Methods:

• Anesthesiologists proficient in English and Turkish served as expert evaluators.
• Ten frequently asked anesthesia questions were selected and translated.
• Responses from ChatGPT 4.0 and DeepSeek V3 in both languages were assessed for overall, content, and communication quality.

Main Results:

• ChatGPT 4.0's English responses were significantly superior to DeepSeek V3's English responses (P<0.001).
• Both ChatGPT 4.0 and DeepSeek V3 demonstrated significantly better overall, content, and communication quality in English compared to their Turkish responses (P<0.001).
• English responses from both LLMs outperformed their Turkish counterparts across all evaluated metrics.

Conclusions:

• ChatGPT 4.0 exhibits higher overall response quality than DeepSeek V3 in English for anesthesia-related queries.
• English language responses from both evaluated LLMs are superior to their Turkish counterparts for anesthesia patient education.
• Further research is needed to optimize LLM performance in non-English languages for healthcare applications.