Speech extraction from vibration signals based on deep learning

Area of Science:

• Acoustics and Signal Processing
• Machine Learning and Artificial Intelligence
• Structural Dynamics and System Identification

Background:

• Traditional speech extraction from vibration signals is sensitive to deviations in model parameters, noise, and boundary conditions.
• Existing methods face challenges in accurately identifying speech characteristics from complex vibroacoustic systems.
• Developing robust methods for speech extraction is crucial for applications in structural health monitoring and audio forensics.

Purpose of the Study:

• To propose and evaluate a deep learning-based approach for extracting speech signals from vibration response data.
• To compare the performance of fully connected and convolutional neural networks for this task.
• To assess the robustness of the proposed method against various deviations, including position, noise, and boundary conditions.

Main Methods:

• Established a vibroacoustic coupling finite element model with voice signals as the excitation source.
• Utilized vibration acceleration signals from response points for deep learning model training, extracting spectral characteristics.
• Trained and tested fully connected and convolutional neural networks, converting extracted signals back to the time domain using amplitude spectra and phase information.

Main Results:

• Fully connected networks exhibited faster convergence rates and better speech extraction quality compared to convolutional networks.
• The proposed method demonstrated robustness to deviations in vibration response point positions and boundary conditions.
• Speech signal noise significantly impacted extraction quality more than vibration signal noise; performance degraded when both were noisy.

Conclusions:

• The deep learning approach offers a robust and effective solution for speech extraction from vibration signals, surpassing traditional methods in handling deviations.
• Fully connected networks are particularly well-suited for this vibroacoustic system identification task.
• The method shows promise for reliable speech recognition in environments with varying signal quality and system parameters.