Simultaneous interpreting with auto-subtitling: Investigating viewer cognitive effort, stress, and comprehension

Area of Science:

• Cognitive Science
• Audiovisual Translation
• Human-Computer Interaction

Background:

• Simultaneous interpreting (SI) is crucial for real-time communication in multilingual settings.
• AI-powered automatic subtitles aid cross-language understanding on virtual platforms.
• Limited research compares SI, auto-subtitling, and their combined effects on users.

Purpose of the Study:

• To compare the effectiveness of auto-subtitling, SI, and a dual-modality approach.
• To evaluate impacts on comprehension, cognitive load, and stress levels.
• To explore user experience with different interlingual translation modes.

Main Methods:

• Mandarin Chinese speakers watched an Arabic video presentation.
• Three groups used: auto-subtitles (Group A), SI (Group B), or both (Group C).
• Electroencephalography (EEG) and comprehension tests measured outcomes.

Main Results:

• No significant differences in content comprehension were found across groups.
• Auto-subtitling alone led to the poorest experience and highest stress.
• SI alone resulted in the highest cognitive effort; the combined approach showed the lowest cognitive effort and stress.

Conclusions:

• Dual-modality systems combining AI subtitles and SI offer significant advantages.
• This integrated approach enhances accessibility and user experience.
• Findings provide insights for integrating AI in simultaneous interpreting.