Language-Specific Differences in Large Language Model Diagnostic Reasoning: A Translation-Controlled Clinical...

Area of Science:

• Artificial Intelligence in Medicine
• Natural Language Processing
• Clinical Decision Support

Background:

• Large language models (LLMs) are increasingly assessed for clinical diagnostic tasks.
• LLM performance may differ significantly across languages, impacting their global applicability.
• Evaluating language influence is vital for safe deployment in diverse healthcare settings.

Purpose of the Study:

• To investigate if input language affects LLM diagnostic reasoning in clinical vignettes.
• To provide insights for multilingual predeployment evaluations of LLMs in non-English healthcare systems.

Main Methods:

• A translation-controlled in silico study using 30 clinical vignettes in English and Polish.
• Six LLMs were evaluated using a structured reflection framework with physician raters.
• Analysis of 720 rater-level evaluations and 360 model-language-vignette responses.

Main Results:

• Language significantly impacted LLM performance, varying by model; some models performed better in English.
• Differences were most pronounced in differential diagnosis quality and examination planning, not just final diagnosis.
• Language robustness was not consistent across evaluated LLMs, with performance gaps in safety-relevant reasoning domains.

Conclusions:

• Multilingual clinical performance of LLMs is highly model-dependent.
• Language-specific evaluations are essential before deploying LLMs in non-English healthcare systems.
• Performance disparities highlight the need for tailored validation to ensure clinical safety and effectiveness.