Evaluation of Commercially Available Machine Interpretation Applications for Simple Clinical Communication

Area of Science:

• Medical Informatics
• Clinical Communication
• Natural Language Processing

Background:

• Accessing professional medical interpreters can be difficult for brief, low-risk clinical exchanges.
• Machine translation (MT) offers potential for verbal communication but lacks thorough evaluation.

Purpose of the Study:

• To evaluate the quality of three MT applications for English-Spanish and English-Mandarin two-way interpretation.
• To compare MT quality against professional human medical interpreters for low-complexity clinical communication.

Main Methods:

• Audio-taped clinical phrases were interpreted using human interpreters and three MT applications.
• Bilingual assessors rated MT interpretation quality (accuracy, fluency, meaning, clinical risk) on 5-point Likert scales.
• A non-inferiority design with a 15% margin assessed MT performance against professional interpreters as the gold standard.

Main Results:

• Acceptable MT interpretation rates for English to Spanish ranged from 68% to 84%; for English to Mandarin, 62% to 76%.
• Acceptable MT interpretation rates for Spanish/Mandarin to English were low, ranging from 36% to 41%.
• None of the MT applications met the non-inferiority threshold compared to human interpreters.

Conclusions:

• MT quality for two-way clinical communication is currently poor, particularly for non-English to English translation.
• Clinicians should prioritize access to professional medical interpreters and avoid relying on current MT tools for patient care.