Relationships between speech-in-noise comprehension, lip reading ability and cortical speech tracking in talking...

In noisy listening environments, visual cues from a speaker's face can significantly boost speech comprehension. The underlying audiovisual integration in the brain involves tracking of audiovisual speech features in the temporal lobe. Moreover, lip reading in silence is associated with tracking of the speech envelope in the low-delta frequency band (0.5-1 Hz). Recently, digital avatars have emerged that can support speech comprehension. Yet, it remains unclear how the human brain integrates such artificial visual signals with natural speech. Here, we employed magnetoencephalography (MEG) to measure the neural responses to a natural video, an avatar generated by deep neural networks, and a degraded video serving as a control. We demonstrate that the avatar can enhance speech-in-noise comprehension to a similar degree as the degraded video, although less than the natural video. We further identify a late response at 600 ms of the neural tracking in the high delta band (1-4 Hz) that predicts audiovisual speech comprehension. In contrast, we did not find the neural tracking in the theta band to be involved in audiovisual integration. Interestingly, we further found that neural tracking in the low delta band was related to silent lip-reading performance. Moreover, the tracking in the low delta band evoked by the avatars was much weaker and occurred earlier than that elicited by the other audiovisual stimuli. Our results show that audiovisual integration can take place even later than suggested in previous work and can index speech understanding. Additionally, the low delta band and the high delta band play distinct roles in visual-only and audiovisual speech processing, suggesting potential avenues for further boosting the abilities of avatars to support speech comprehension.