Singing to speech conversion with generative flow

Area of Science:

• Artificial Intelligence
• Speech Processing
• Machine Learning

Background:

• Singing to speech (S2S) conversion is a novel cross-domain task.
• Existing methods lack naturalness and phonetic accuracy.
• Deep learning approaches offer potential for S2S advancement.

Purpose of the Study:

• Introduce the first deep learning-based S2S system.
• Transform singing into speech, preserving phonetic content.
• Enhance naturalness and phonetic similarity in converted speech.

Main Methods:

• Developed a generative flow model inspired by Glow-TTS.
• Adapted monotonic alignment search (MAS) for S2S.
• Utilized a duration predictor to handle modality differences.

Main Results:

• The proposed deep learning model outperforms signal processing baselines in naturalness.
• It surpasses transcribe-and-synthesize methods in phonetic similarity.
• S2S conversion proves effective for low-resource lyrics transcription.

Conclusions:

• The novel deep learning S2S system achieves high naturalness and phonetic accuracy.
• This technology offers a viable solution for voice conversion challenges.
• S2S conversion can serve as a valuable data augmentation technique.