Related Experiment Video
Updated: Jan 18, 2026
06:02
Evaluating Usability Aspects of a Mixed Reality Solution for Immersive Analytics in Industry 4.0 Scenarios
Published on: October 6, 2020
2.6K
A Robust Bilinear Framework for Real-Time Speech Separation and Dereverberation in Wearable Augmented Reality
Alon Nemirovsky1, Gal Itzhak1, Israel Cohen1
1Andrew and Erna Viterbi Faculty of Electrical & Computer Engineering, Technion-Israel Institute of Technology, Haifa 3200003, Israel.
Sensors (Basel, Switzerland)
|September 13, 2025
Summary
This study introduces a new audio enhancement framework for wearable augmented reality (AR) devices. It improves speech clarity in noisy environments by separating sources and reducing echo, making AR experiences more robust.
Area of Science:
- Audio Signal Processing
- Augmented Reality Systems
- Acoustic Engineering
Background:
- Wearable augmented reality (AR) devices require robust audio processing in dynamic acoustic environments.
- Real-time speech enhancement is crucial for effective AR user interaction.
- Existing methods struggle with localization errors and reverberation in AR scenarios.
Purpose of the Study:
- To develop a low-complexity, real-time bilinear framework for speech source separation and dereverberation in AR.
- To enhance robustness against direction-of-arrival (DOA) estimation errors using a region-of-interest (ROI) beamformer.
- To introduce a multi-constraint beamforming design for improved source separation and noise reduction.
Main Methods:
- A bilinear framework for real-time speech enhancement.
- Implementation of a region-of-interest (ROI) beamformer to address DOA estimation errors.
- Development of a multi-constraint beamforming approach for simultaneous source preservation or suppression.
- Validation using the Speech Enhancement for Augmented Reality (SPEAR) Challenge dataset and real-world recordings.
Main Results:
- ROI-based steering significantly improves robustness to localization errors while maintaining noise and reverberation suppression.
- ROI beamforming introduces increased high-frequency leakage from desired and undesired sources.
- Multi-constraint beamforming enhances source separation with a minor reduction in noise suppression.
- The integrated framework demonstrates practical efficiency for real-time audio enhancement.
Conclusions:
- The proposed bilinear framework with ROI and multi-constraint beamforming offers an efficient solution for real-time audio enhancement in wearable AR.
- The ROI beamformer enhances robustness to localization inaccuracies inherent in AR head movements.
- The multi-constraint design provides flexibility in managing multiple sound sources within the AR audio scene.
