一个多模式的动态变化自编码器,用于视听语音表示学习
Samir Sadok1, Simon Leglaive1, Laurent Girin2
1CentraleSupélec IETR UMR CNRS 6164, France.
概括
本研究介绍了一种多式和动态变量自编码器 (MDVAE),用于无监督的视听语音表现学习. MDVAE有效地解和结合视听信息,以改善情绪识别.
科学领域:
- 机器学习 机器学习
- 人工智能的人工智能
- 信号处理 信号处理
背景情况:
- 像语音这样的高维数据具有潜在的规律性,暗示了低维的潜在表示.
- 深潜变量生成模型,特别是变量自编码器 (VAE),对于无监督表示学习是有效的.
- 现有的VAE已经扩展到多式联络和序列数据,但需要专门的视听语音模型.
研究的目的:
- 开发一种新型的多模态和动态变量自编码器 (MDVAE),用于无监督的视听语音表示学习.
- 构建潜伏空间,以解开视听语言中的静态,动态,模式特定和模式共同的因素.
- 评估MDVAE在将视听信息与其应用于情绪识别方面的有效性.
主要方法:
- 采用了两阶段的无监督培训方法,从每个模式的独立矢量量化VAE (VQ-VAE) 开始.
- 第二阶段包括对MDVAE进行中介表示的培训,以分离静态/动态和共享/特定信息.
- 实验包括视听语音操纵,面部图像否定和使用学习的潜伏表征来识别情绪.
主要成果:
- MDVAE成功地学会了视听语言的综合隐藏表现,有效地整合了音频和视觉信息.
- 解散的潜伏空间允许分离静态,动态,模式特定和模式常见的因素.
- 静态潜伏表示在有限的标记数据下实现了情感识别的高精度,超过了单模和基于变压器的模型.
结论:
- 拟议的MDVAE为无监督的视听语音表现学习提供了一个强大的框架.
- 该模型能够解开各种潜在因素,从而提高对视听语音数据的理解和操纵.
- 对于下游任务,例如情绪识别,MDVAE显示出显著的潜力,特别是在低数据的系统中.
相关概念视频
01:22Multi-input and Multi-variable systems
106
Cruise control systems in cars are designed as multi-input systems to maintain a driver's desired speed while compensating for external disturbances such as changes in terrain. The block diagram for a cruise control system typically includes two main inputs: the desired speed set by the driver and any external disturbances, such as the incline of the road. By adjusting the engine throttle, the system maintains the vehicle's speed as close to the desired value as possible.
In the absence...
In the absence...
106
01:27State Space Representation
208
The frequency-domain technique, commonly used in analyzing and designing feedback control systems, is effective for linear, time-invariant systems. However, it falls short when dealing with nonlinear, time-varying, and multiple-input multiple-output systems. The time-domain or state-space approach addresses these limitations by utilizing state variables to construct simultaneous, first-order differential equations, known as state equations, for an nth-order system.
Consider an RLC circuit, a...
Consider an RLC circuit, a...
208
01:27Associative Learning
370
Associative learning is a fundamental concept in behavioral psychology, wherein a connection is established between two stimuli or events, leading to a learned response. This process is critical in understanding how behaviors are acquired and modified. Conditioning, the mechanism through which associations are formed, can be divided into two main types: classical conditioning and operant conditioning, each elucidating different aspects of associative learning.
Classical conditioning, also known...
Classical conditioning, also known...
370
01:21Perceiving Loudness, Pitch, and Location
212
The human brain perceives pitch through two primary mechanisms reflected in place theory and frequency theory. Each mechanism describes how sound waves are interpreted as specific pitches by the brain, offering insights into the intricate processes of auditory perception.
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by...
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by...
212
01:16Control Volume and System Representations
1.2K
Two key frameworks are employed to analyze mass, energy, and momentum transfer: the control volume approach and the system approach. These frameworks offer different perspectives, depending on whether the focus is on a specific region in space (control volume approach) or a defined mass of fluid (system approach).
The control volume approach considers a stationary region in space through which fluid flows. This region is bounded by a control surface. For instance, in the case of water...
The control volume approach considers a stationary region in space through which fluid flows. This region is bounded by a control surface. For instance, in the case of water...
1.2K
01:22Chunking and Rehearsal in Sensory Memory
213
Improving short-term memory can be achieved through techniques like chunking and rehearsal. Chunking involves organizing information into larger, more manageable units. This technique is particularly useful for information that exceeds the typical memory span of between five and nine items. For instance, logging into an online account with a password like "ta89vq0179gz" involves grouping letters and numbers into three chunks—ta89, vq01, and 79gz. It makes large amounts of...
213