SpeechPalette: A Comprehensive Speech Editing Method for Text-Based Speech Editing, One-Shot TTS and Attributes...

Area of Science:

• Artificial Intelligence
• Speech Processing
• Machine Learning

Background:

• Existing speech editing systems demand significant manual input and offer limited attribute control.
• These limitations hinder the practical application of speech editing technologies.

Purpose of the Study:

• To introduce SpeechPalette, a novel speech editing method for high-quality, user-controlled modification of speech attributes.
• To enable fine-grained control over speech segments, including text, pitch, and duration.

Main Methods:

• A decoupling approach disentangles speech attributes (text, pitch, duration) from the input audio.
• A mask and prediction mechanism reconstructs speech, enhanced by a diffusion model predicting residual errors.
• The system demonstrates capabilities in text-based editing, pitch/speed adjustments, and one-shot text-to-speech.

Main Results:

• SpeechPalette achieves superior performance in various speech editing tasks compared to baseline systems.
• The method provides explicit, fine-grained control over speech attributes.
• Experimental results and ablation studies validate the effectiveness of the proposed approach.

Conclusions:

• SpeechPalette presents a balanced approach to speech editing, offering high quality, explicit control, and practical deployment.
• The method outperforms similarly-sized systems and demonstrates versatility across multiple speech manipulation tasks.