StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech...

Area of Science:

• Artificial Intelligence
• Speech Technology
• Machine Learning

Background:

• Text-to-speech (TTS) synthesis aims to generate human-like speech from text.
• Previous TTS models often require reference speech for style control or lack naturalness.
• Large speech language models (SLMs) show promise in enhancing speech synthesis quality.

Purpose of the Study:

• To develop a novel text-to-speech (TTS) model, StyleTTS 2, capable of human-level speech synthesis.
• To improve speech naturalness and style control in TTS without requiring reference speech.
• To demonstrate the effectiveness of style diffusion and adversarial training with SLMs in TTS.

Main Methods:

• StyleTTS 2 utilizes style diffusion to model speech styles as latent variables, enabling diverse synthesis without reference audio.
• The model incorporates large pre-trained SLMs (e.g., WavLM) as discriminators for adversarial training.
• Novel differentiable duration modeling is employed for end-to-end training, enhancing speech naturalness.

Main Results:

• StyleTTS 2 achieved human-level performance on the single-speaker LJSpeech dataset, surpassing human recordings.
• The model matched human performance on the multispeaker VCTK dataset.
• StyleTTS 2 demonstrated superior zero-shot speaker adaptation performance on the LibriTTS dataset compared to prior models.

Conclusions:

• StyleTTS 2 represents a significant advancement in TTS, achieving the first human-level synthesis on both single and multispeaker datasets.
• The combination of style diffusion and adversarial training with large SLMs is highly effective for natural and controllable TTS.
• The model's open availability of audio demos and source code facilitates further research and application in speech synthesis.