Evaluating a Large Language Model in Translating Patient Instructions to Spanish Using a Standardized Framework.

Area of Science:

• Medical Informatics
• Natural Language Processing
• Health Equity

Background:

• Patients with limited English proficiency face significant barriers accessing crucial written medical information.
• Effective communication in healthcare is vital for patient safety and adherence to treatment plans.
• Large language models (LLMs) offer potential solutions for translating patient materials, but require rigorous validation.

Purpose of the Study:

• To evaluate the quality of Spanish translations of personalized pediatric patient instructions generated by GPT-4o.
• To compare GPT-4o's translation quality against professional human translators using a standardized framework.
• To assess the potential of LLMs in improving access to language-concordant healthcare information.

Main Methods:

• A cross-sectional study comparing GPT-4o and professional human translations of 20 pediatric patient instructions.
• Translations were evaluated by three independent medical translators using the Multidimensional Quality Metrics (MQM) framework.
• Equivalence testing was performed to compare translation quality scores.

Main Results:

• GPT-4o translations demonstrated comparable quality to human translations, with no statistically significant difference (mean difference 1.6 MQM points).
• The LLM produced fewer mistranslation errors.
• Professional translators preferred GPT-4o translations in 52% of cases.

Conclusions:

• GPT-4o can generate high-quality Spanish translations of patient instructions, meeting clinical standards.
• LLMs show promise in reducing translation workload, potentially freeing resources for other languages.
• Continued human oversight is essential, but LLMs can enhance efficiency in healthcare translation.