Accuracy of Artificial Intelligence vs Professionally Translated Discharge Instructions

Area of Science:

• Medical Informatics
• Health Services Research
• Clinical Translation

Background:

• Patients with limited English proficiency face significant barriers to accessing translated health information, potentially leading to poorer health outcomes.
• Artificial intelligence (AI) offers a potential solution for improving access to translated clinical materials, but its accuracy in routine practice needs evaluation.

Purpose of the Study:

• To compare the accuracy and quality of AI-generated translations versus professional human translations for pediatric inpatient discharge instructions.

Main Methods:

• A comparative effectiveness analysis evaluated AI (Azure) and professional human translations of pediatric discharge instructions in Simplified Chinese, Somali, Spanish, and Vietnamese.
• Two professional translators assessed translations for fluency, adequacy, meaning, and error severity on a 5-point scale, blinded to the translation source.

Main Results:

• AI translations were generally inferior to professional translations across all tested languages, with lower scores for fluency, adequacy, and meaning.
• Spanish AI translations were non-inferior in adequacy and error severity but inferior in fluency and meaning compared to professional translations.
• AI translations for Chinese, Vietnamese, and Somali languages demonstrated inferiority across all evaluated metrics, with Somali showing the most significant differences.

Conclusions:

• AI translation of discharge instructions performed comparably to professional translation for Spanish but was worse for Chinese, Somali, and Vietnamese.
• Implementing AI-based translation requires careful consideration of languages with lesser diffusion to avoid exacerbating health inequities.