Spectral Flux-Based Convolutional Neural Network Architecture for Speech Source Localization and Its Real-Time...

Area of Science:

• Acoustics and Signal Processing
• Artificial Intelligence and Machine Learning

Background:

• Speech source localization (SSL) is crucial for audio processing but challenging in realistic noisy and reverberant conditions.
• Existing SSL algorithms often struggle with performance degradation in complex acoustic environments.

Purpose of the Study:

• To develop a real-time, robust CNN-based SSL algorithm capable of handling realistic background acoustic conditions.
• To evaluate the algorithm's performance on a prototype platform for practical applications.

Main Methods:

• Utilized a convolutional neural network (CNN) trained with features derived from the imaginary-real coefficients of the short-time Fourier transform (STFT) and Spectral Flux (SF).
• Employed delay-and-sum (DAS) beamforming as part of the input feature extraction process.
• Trained the CNN model using diverse noisy speech recordings and tested on unseen acoustic environments.

Main Results:

• The proposed CNN-SSL algorithm demonstrated significant improvements over five previously published methods under various noisy conditions.
• Achieved high accuracy (89.68% at 5dB SNR under Babble noise) with low latency (21 ms per frame).
• Successfully implemented and tested for real-time operation on a Raspberry Pi prototype.

Conclusions:

• The integration of Spectral Flux (SF) with beamforming enhances the CNN's ability to learn temporal variations in speech spectra, leading to improved SSL performance.
• The developed algorithm offers a robust and efficient solution for real-time SSL, suitable for portable, battery-operated devices.
• This work has significant implications for enhancing audio processing in smart loudspeakers and hearing improvement devices.