Breaking language barriers: evaluating AI translation for urogynecology patient education

Patient education materials (PEMS) are not always available in a patients' written language. Google Translate, using artificial intelligence (AI), provides a cost-effective, widely available translation option. We evaluated understandability and actionability of Google AI translations of International Urogynecological Association (IUGA)'s midurethral sling (MUS) leaflet with the Patient Education Materials Assessment Tool (PEMAT). We also evaluated translation equivalence using the comparability of language (CL) and similarity of interpretability (SI) assessments. We hypothesized no differences between back-translated and original version's PEMAT scores, CL, or SI and that PEMAT back-translated version scores would be non-inferior to original versions.

IUGA's MUS leaflet was translated into Mandarin Chinese and Swahili, then back-translated into English using Google Translate. Participants scored original and back-translated versions using the PEMAT (0-100, better scores are higher) and CL and SI (1-7 scale, lower scores better). Participants were randomized whether they evaluated the original or back-translated version first to decrease bias.

No significant differences were found on understandability or actionability between the original English and back-translated Mandarin Chinese or Swahili versions. However, neither met the pre-specified non-inferiority margin of 5-points on the PEMAT. CL (mean: 4.4±0.9) and SI (mean: 4±1.2) scores for both indicated moderate comparability and similarity.

Original and back-translated versions of Mandarin Chinese or Swahili MUS IUGA leaflets did not score differently on the PEMAT although we did not meet the non-inferiority margin. CL and SI support Google translate as a moderately useful tool in providing IUGA leaflets in languages not commonly available.