隐藏歌手:通过神经音频编码器和潜在扩散模型高质量的歌声合成
Ji-Sang Hwang1, Sang-Hoon Lee2, Seong-Whan Lee1
1Department of Artificial Intelligence, Korea University, 02841, Seoul, Republic of Korea.
概括
隐藏Singer使用神经音频编解码器和潜在扩散模型来生成高质量的歌唱声音. 这种新的方法克服了当前唱歌语音合成 (SVS) 系统的局限性,即使在未标记的数据上进行训练.
科学领域:
- 人工智能的人工智能
- 语音合成 语音合成
- 机器学习 机器学习
背景情况:
- 无声扩散模型在生成任务中显示出希望,但在复杂的,时间变化的音频合成中面临挑战.
- 歌声合成 (SVS) 需要高维样本和长期的声学特征,这给现有的扩散模型带来了复杂性问题.
研究的目的:
- 提出HiddenSinger,一个高质量的SVS系统,解决复杂性和可控制性限制.
- 引入一个无监督学习框架,HiddenSinger-U,用于用未标记的数据训练SVS模型.
主要方法:
- 一个带有自动编码器的神经音频编解码器将音频压缩成一个低维的潜向量,用于高保真重建.
- 隐性扩散模型被用来从乐谱中取样隐性表示.
- 隐藏Singer-U框架允许使用未标记的歌声数据集进行训练.
主要成果:
- 与之前的SVS模型相比,HiddenSinger表现出优越的音频质量.
- 隐藏Singer-U成功地合成了高质量的歌声,这些歌声完全是在未标记的数据上训练的.
结论:
- 隐藏Singer提供了一个强大的解决方案,用于高质量,可控制的唱歌语音合成.
- 无监督的HiddenSinger-U框架通过利用未标记的数据集来扩大SVS模型的适用性.
相关概念视频
Downsampling
134
When considering a sampled sequence with zero values between sampling instants, one can replace it by taking every N-th value of the sequence. At these integer multiples of N, the original and sampled sequences coincide. This process, known as decimation, involves extracting every N-th sample from a sequence, thereby creating a more efficient sequence.
The Fourier transform of the decimated sequence reveals a combination of scaled and shifted versions of the original spectrum. This...
The Fourier transform of the decimated sequence reveals a combination of scaled and shifted versions of the original spectrum. This...
134
Perceiving Loudness, Pitch, and Location
198
The human brain perceives pitch through two primary mechanisms reflected in place theory and frequency theory. Each mechanism describes how sound waves are interpreted as specific pitches by the brain, offering insights into the intricate processes of auditory perception.
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by...
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by...
198
Auditory Perception
322
The auditory system is essential for sound perception, utilizing various critical structures. When sound waves enter the outer ear, they travel through the ear canal and cause the eardrum to vibrate. These vibrations are then transmitted to the middle ear, where three tiny bones – the malleus, incus, and stapes – amplify the sound. This amplification is crucial, as it ensures that the sound vibrations are strong enough to be conveyed to the inner ear. These vibrations then reach the...
322
Larynx
1.3K
The human larynx, often referred to as the voice box, is an intricate organ located in the neck. It serves as a pathway for air to enter the lungs during respiration and is an essential component of voice production.
Anatomy of the Larynx
The larynx consists of various components, including cartilage, muscles, and vocal cords. Its structure includes three large unpaired cartilages—the thyroid, cricoid, and epiglottis—and three smaller paired cartilages—the arytenoids,...
Anatomy of the Larynx
The larynx consists of various components, including cartilage, muscles, and vocal cords. Its structure includes three large unpaired cartilages—the thyroid, cricoid, and epiglottis—and three smaller paired cartilages—the arytenoids,...
1.3K
Pulse amplitude and quality
1.7K
Pulse amplitude is a crucial indicator of cardiac health because it provides valuable insights into the strength of left ventricular contractions and the overall uniformity of blood circulation within the vasculature. The strength of the pulse is directly related to the force with which the heart contracts and the volume of blood being pumped.
A weak or absent pulse may indicate reduced cardiac output or poor left ventricular contraction, which can be signs of cardiovascular dysfunction or...
A weak or absent pulse may indicate reduced cardiac output or poor left ventricular contraction, which can be signs of cardiovascular dysfunction or...
1.7K
Auditory Pathway
5.3K
Auditory pathways constitute the complex neural circuits responsible for transmitting and interpreting auditory information from the peripheral auditory system to the brain. Sound waves are initially captured by the outer ear, funneled through the ear canal, and reach the tympanic membrane (eardrum). These vibrations are transmitted via the middle ear's ossicles to the inner ear's cochlea.
When viewed cross-sectionally, the cochlea reveals the scala vestibuli and scala tympani flanking...
When viewed cross-sectionally, the cochlea reveals the scala vestibuli and scala tympani flanking...
5.3K


