Multisensory integration with a head-mounted display: sound delivery and self-motion

We tested whether the method of sound delivery affects people's ability to integrate information from multiple modalities when they are walking and using a head-mounted display (HMD).

HMDs increasingly support mobile work. Human operators may benefit from auditory support when using an HMD. However, it is unclear whether sound is better delivered publicly in free field or privately via earpiece and what the effect of walking is.

Participants identified mismatches between sounds and visual information on an HMD. Participants heard the sounds via either earpiece or free field while they either sat or walked about the test room.

When using an earpiece, participants performed the mismatch task equally well whether sitting or walking, but when using free-field sound, participants performed the task significantly worse when walking than when sitting (p = .006).

The worse performance for participants using free-field sound while walking may relate to spatial and motion inconsistencies between visual events on the head-referenced HMD and auditory events from world-referenced speakers. Researchers should more frequently examine the effect of self-motion on people's ability to perform various multisensory tasks.

When multisensory integration tasks are performed with an HMD and free-field delivery of sound, as may happen in medicine, transportation, or industry, performance may suffer when the relative location of sound changes as the user moves.