A multimedia speech corpus for audio visual research in virtual reality (L)

Area of Science:

• Perceptual Psychology
• Virtual Reality Research
• Acoustics

Background:

• Virtual reality (VR) offers novel ways to study perception.
• Creating realistic auditory-visual stimuli in VR is challenging.
• Existing resources for VR perceptual research are limited.

Purpose of the Study:

• To present a publicly available database of anechoic audio speech samples with matching stereoscopic and 360° video.
• To provide tools for creating complex VR simulations with multiple, spatially-placed talkers.
• To facilitate advanced perceptual research within virtual environments.

Main Methods:

• Recording anechoic audio speech samples.
• Capturing synchronized stereoscopic and 360° video.
• Developing software for VR environment integration.
• Describing recording conditions, corpus contents, and usage guidelines.

Main Results:

• A comprehensive database of audio-visual speech samples is now available.
• The accompanying software allows flexible simulation of multiple talkers in VR.
• The materials support the creation of ecologically valid, albeit contrived, perceptual scenarios.

Conclusions:

• This database and toolset significantly advance the capabilities for perceptual research in virtual reality.
• It enables the creation of sophisticated, multi-talker auditory-visual simulations.
• Researchers can now explore complex perceptual phenomena in controlled VR settings.