HiddenSinger: High-quality singing voice synthesis via neural audio codec and latent diffusion models

Area of Science:

• Artificial Intelligence
• Speech Synthesis
• Machine Learning

Background:

• Denoising diffusion models show promise in generative tasks but face challenges in complex, time-varying audio synthesis.
• Singing voice synthesis (SVS) requires high-dimensional samples and long-term acoustic features, posing complexity issues for existing diffusion models.

Purpose of the Study:

• To propose HiddenSinger, a high-quality SVS system addressing complexity and controllability limitations.
• To introduce an unsupervised learning framework, HiddenSinger-U, for training SVS models with unlabeled data.

Main Methods:

• A neural audio codec with an autoencoder compresses audio into a low-dimensional latent vector for high-fidelity reconstruction.
• Latent diffusion models are employed to sample latent representations from musical scores.
• The HiddenSinger-U framework enables training using unlabeled singing voice datasets.

Main Results:

• HiddenSinger demonstrates superior audio quality compared to previous SVS models.
• HiddenSinger-U successfully synthesizes high-quality singing voices trained exclusively on unlabeled data.

Conclusions:

• HiddenSinger offers a robust solution for high-quality, controllable singing voice synthesis.
• The unsupervised HiddenSinger-U framework expands the applicability of SVS models by leveraging unlabeled datasets.