Incorporating Virtual Reality Agents During a Dichotic Speech Reception Task: Insights From the Heart

Listening effort is moderated by not only task difficulty, but also success importance. In real communication scenarios, success importance varies based upon the social context. However, in the laboratory, it can be challenging to manipulate social context without compromising experimental control. Outside of hearing sciences, studies have applied virtual reality (VR) to incorporate social context in a controlled and repeatable manner. Several of these studies have demonstrated that social manipulations in VR can reliably elicit changes in cardiovascular measures. Here, we investigated the effect of adding VR agents to a speech reception task, while measuring from the cardiovascular system.

Twenty-eight, normally hearing participants undertook a dichotic speech reception task. Sentences in stationary noise were presented dichotically, that is, different sentences presented simultaneously to each ear. Participants were tasked to either repeat one of the sentences (single-sentence condition) or both of the sentences (dual-sentence condition). The task was conducted under two VR conditions: (1) in the presence of agents, who provided sporadic performance feedback and (2) in the presence of nonagent controls, without any feedback given. Alongside task performance, we quantified changes in cardiovascular measures, relative to pretask baselines: heart rate variability, pre-ejection period, heart rate, and blood pressure. After each condition, participants rated their subjective effort, difficulty, performance, and engagement.

Performance and the subjective perception of performance were lower, while subjective effort and difficulty were higher, in the dual-sentence condition, compared with the single-sentence condition. Heart rate was the only cardiovascular measure that was sensitive to the experimental manipulations. Contrary to our expectations, heart rate increased in the nonagent control conditions, compared with the agent conditions. An exploratory analysis revealed heart rate fluctuations within a trial: heart rate was higher during the first 6 sec of the trial (reflecting the presentence masking noise and the sentence presentation) in the dual-sentence condition, compared with the single-sentence condition.

This study was the first to incorporate VR agents who provided performance feedback during a dichotic speech reception task. Our results suggest that the VR agents did not increase success importance, which could be attributed to a lack of realism of the agents. We also demonstrated that the cardiovascular response to experimental manipulations may differ depending on the data window selected for analysis.