Speech Discrimination in Real-World Group Communication Using Audio-Motion Multimodal Sensing

Area of Science:

• Human-Computer Interaction
• Signal Processing
• Communication Systems

Background:

• Speech discrimination is vital for analyzing human verbal communication.
• Simultaneous speakers in dynamic group settings pose challenges for audio-only speech detection.
• Existing methods struggle to accurately distinguish individual speech in multi-participant environments.

Purpose of the Study:

• To investigate if combining audio and physical motion data improves speech discrimination.
• To assess the effectiveness of multimodal sensing using wearable smartphones for utterance detection.
• To enhance speech discrimination in dynamic group communication scenarios.

Main Methods:

• Recorded utterance and physical activity data from university students using neck-worn smartphones.
• Analyzed the temporal relationship between physical activities and speech utterances.
• Trained and tested audio-only, motion-only, and combined audio-motion classifiers for speech discrimination.

Main Results:

• Physical activities across a wide frequency range were found to co-occur with speech utterances.
• The audio-motion classifier achieved higher accuracy (92.2%) compared to audio-only (80.4%) and motion-only (87.8%) for intra-participant classification.
• The combined classifier also outperformed single-modality classifiers in inter-individual classification (83.2% vs. 67.7% and 71.9%).

Conclusions:

• Multimodal sensing combining audio and motion data effectively improves speech discrimination.
• Widely available smartphones can be utilized for robust utterance detection in dynamic group communications.
• This approach offers a promising solution for analyzing complex social interactions and verbal communication.