用文本和语音驱动的手势动画进行零拍摄风格转移,使用多式风格编码的对抗性解
Mireille Fares1, Catherine Pelachaud2, Nicolas Obin3
1The Institute of Intelligent Systems and Robotics (ISIR), Sciences et Technologies de la Musique et du Son (STMS), Sorbonne University, Paris, France.
Frontiers in artificial intelligence
|June 28, 2023
概括
本研究介绍了一种机器学习模型,用于利用语音内容和演讲风格合成虚拟代理手势. 该模型可以实现零拍摄多式模式风格传输,为看不见的扬声器生成现实的手势.
科学领域:
- 人工智能的人工智能
- 人与计算机的交互
- 机器学习 机器学习
背景情况:
- 个性化人与代理的互动需要模拟虚拟代理的行为风格.
- 现有的方法可能难以适应不同的演讲风格,特别是那些在培训中没有遇到的演讲风格.
研究的目的:
- 开发一种高效的机器学习方法来合成由语音内容和演讲风格驱动的类似人类的手势.
- 为了实现零拍摄多式模式风格转移,允许在训练期间看不见的扬声器风格的手势生成.
主要方法:
- 一种使用扬声器风格编码器网络和序对序合成网络的新方法.
- 该模型将内容 (Mel谱图,文本语义) 与风格 (扬声器嵌入) 分离,用于手势生成.
- 通过对目标扬声器风格嵌入条件化手势合成来实现零射击多式模式风格传输.
主要成果:
- 该模型成功地基于语音内容合成了手势,并从看不见的扬声器转移了扬声器风格.
- 客观和主观的评估验证了模型在准确的风格转移下产生手势的能力.
- 这种方法证明了学习高质量的扬声器表示,用于手势合成.
结论:
- 拟议的机器学习模型有效地将手势与多式模式风格传输合成为零拍摄设置.
- 内容和风格的分离对于适应新型说话者行为至关重要.
- 这项工作通过现实和风格可控的虚拟代理手势来推进个性化的人与代理的互动.
相关概念视频
Masking and Demasking Agents
2.5K
EDTA titrations may necessitate masking and demasking agents to temporarily protect a particular metal ion in a mixture from the EDTA reaction. These agents facilitate the sequential analysis of the metal ions by forming stable complexes with some—but not all—metal ions during certain steps.
There are many masking agents, such as cyanide, fluoride, triethanolamine, thiourea, and 2,3-bis(sulfanyl)propan-1-ol (formerly 2,3-dimercapto-1-propanol), with the masking agent chosen based on...
There are many masking agents, such as cyanide, fluoride, triethanolamine, thiourea, and 2,3-bis(sulfanyl)propan-1-ol (formerly 2,3-dimercapto-1-propanol), with the masking agent chosen based on...
2.5K
Multi-input and Multi-variable systems
132
Cruise control systems in cars are designed as multi-input systems to maintain a driver's desired speed while compensating for external disturbances such as changes in terrain. The block diagram for a cruise control system typically includes two main inputs: the desired speed set by the driver and any external disturbances, such as the incline of the road. By adjusting the engine throttle, the system maintains the vehicle's speed as close to the desired value as possible.
In the absence...
In the absence...
132
Force Classification
1.3K
Forces play a crucial role in the study of physics and engineering. They are essential in describing the motion, behavior, and equilibrium of objects in the physical world. Forces can be classified based on their origin, type, and direction of action.
Contact and non-contact forces are two of the most widely used categories of forces. As the name suggests, contact forces require physical contact between two objects to act upon each other. Examples of contact forces include frictional,...
Contact and non-contact forces are two of the most widely used categories of forces. As the name suggests, contact forces require physical contact between two objects to act upon each other. Examples of contact forces include frictional,...
1.3K
Position and Displacement Vectors
9.6K
To describe the motion of an object, one should first be able to describe its position (where it is at any particular time). More precisely, the position needs to be specified relative to a convenient frame of reference. A frame of reference is an arbitrary set of axes from which the position and motion of an object are described. Earth is often used as a frame of reference to describe the position of an object in relation to stationary objects on Earth.
Further, several important kinds of...
Further, several important kinds of...
9.6K
State Space to Transfer Function
243
The conversion of state-space representation to a transfer function is a fundamental process in system analysis. It provides a method for transitioning from a time-domain description to a frequency-domain representation, which is crucial for simplifying the analysis and design of control systems.
The transformation process begins with the state-space representation, characterized by the state equation and the output equation. These equations are typically represented as:
The transformation process begins with the state-space representation, characterized by the state equation and the output equation. These equations are typically represented as:
243
Transformation
35
Microbial communities are dynamic environments where cell lysis releases free DNA into the surroundings. Other cells can take up this extracellular DNA through a process known as transformation.When a cell incorporates this foreign DNA into its genome, resulting in genetic modification, the process is known as transformation. Cells capable of this process are termed competent. Competence can be natural, as observed in certain bacteria and archaea, or artificially induced in the...
35


