EMBC Special Issue: Advancing Electrolaryngeal Speech Enhancement Through Speech-Text Representation Learning

Area of Science:

• Speech Processing
• Artificial Intelligence
• Assistive Technologies

Background:

• Electrolaryngeal (EL) speech, used by laryngectomees, suffers from distortion, limited phonetic variation, and unnatural prosody, impacting naturalness and intelligibility.
• Existing sequence-to-sequence (seq2seq) voice conversion (VC) models for EL-to-normal speech conversion (EL2SP) face challenges due to mismatches between EL and normal speech, leading to cumulative mapping errors.

Purpose of the Study:

• To develop a novel representation learning framework that integrates speech and text representations to enhance mapping and reconstruction quality in seq2seq VC models for EL2SP.
• To improve the naturalness and intelligibility of electrolaryngeal speech for improved verbal communication.

Main Methods:

• A network incorporating auxiliary text information using pretrained modules to learn integrated speech-text representations.
• An autoencoder-style reconstruction strategy to finalize the EL2SP model, inheriting learned representations without increased complexity.
• Three fusion strategies (middle-, input-, hybrid-level) and an additional reconstruction loss on integrated representations to refine transfer and enhance learning.

Main Results:

• The proposed methods, combined with data augmentation, significantly outperform baseline models relying solely on speech representations in EL2SP conversion quality and intelligibility.
• Progressive improvements were observed with increasing system design depth, validating the effectiveness of the integrated approach.

Conclusions:

• The developed methodology offers a practical and extensible solution for enhancing electrolaryngeal speech.
• This approach holds significant potential for advancing assistive communication technologies for individuals with speech impairments.