语音Act:通过语音产生全身运动
IEEE transactions on visualization and computer graphics
|March 3, 2025
概括
SpeechAct通过使用一种新的混合点表示和对比学习,从语音中产生现实的和多样化的全身运动. 这种方法通过提高运动自然性和独特性来增强计算机图形和虚拟现实应用程序.
科学领域:
- 计算机图形 计算机图形
- 虚拟现实 虚拟现实 虚拟现实
- 人与计算机的交互
背景情况:
- 从语音中产生自然和多样化的全身运动对于沉浸式体验至关重要.
- 现有的方法往往缺乏现实主义和语音驱动的运动生成的多样性.
研究的目的:
- 介绍SpeechAct,一种用于从语音音频中生成现实和多样化的全身运动的新方法.
- 为了提高产生的运动的自然性,平滑性和独特性.
主要方法:
- 使用混合点表示,将3D车身模型的关键点和表面点结合起来.
- 采用VQ-VAE用于运动代码书学习和音频到运动回归的翻译模型.
- 使用负样本实现对比的运动学习,以增强运动多样性.
主要成果:
- 混合点表示方便流和自然的运动生成.
- 对比式学习有效地增加了产生的运动的多样性.
- 集成的面部生成器确保确定性和同步的面部运动.
结论:
- SpeechAct在产生现实和多样化的语音驱动全身运动方面表现出卓越的性能.
- 建议的混合表示和对比学习对提高运动质量是有效的.
- 该方法在计算机图形和沉浸式VR/AR中的应用具有显著的潜力.
相关概念视频
Absolute Motion Analysis- General Plane Motion
199
Visualize a drone, with its propellers spinning rapidly, hovering mid-air. The fascinating movements and operations of this drone can be comprehended by applying the principle of general plane motion.
As the drone's propellers rotate, an upward force is generated that counteracts the force of gravity, enabling the drone to lift off from the ground. This initial movement of the drone is along a straight path, representing a form of translational motion. In this phase, every point on the...
As the drone's propellers rotate, an upward force is generated that counteracts the force of gravity, enabling the drone to lift off from the ground. This initial movement of the drone is along a straight path, representing a form of translational motion. In this phase, every point on the...
199
Relative Motion Analysis - Velocity
334
A stroke engine has a slider-crank mechanism that converts rotational motion from the crank into linear motion of the slider or vice versa. This mechanism consists of three main parts: the crank, the connecting rod, and the slider.
When an external force is exerted, it sets the crank into a rotational movement. This, in turn, instigates the motion of the connecting rod, leading to what is referred to as a general plane motion. This process involves two key points - point A on the connecting rod...
When an external force is exerted, it sets the crank into a rotational movement. This, in turn, instigates the motion of the connecting rod, leading to what is referred to as a general plane motion. This process involves two key points - point A on the connecting rod...
334
Planar Rigid-Body Motion
370
Understanding the movement of a rigid body in planar motion involves recognizing that every particle within this body is traversing a path that maintains a consistent distance from a specific plane. This concept is fundamental in the study of physics and mechanical engineering, and it allows us to comprehend better how objects move in space.
Planar motion is typically divided into three distinct categories. The first is rectilinear translation, demonstrated by a subway train that moves along...
Planar motion is typically divided into three distinct categories. The first is rectilinear translation, demonstrated by a subway train that moves along...
370
Motional Emf
3.1K
Magnetic flux depends on three factors: the strength of the magnetic field, the area through which the field lines pass, and the field's orientation with respect to the surface area. If any of these quantities vary, a corresponding variation in magnetic flux occurs. If the area through which the magnetic field lines are passing changes, then the magnetic flux also changes. This change in the area can be of two types: the flux through the rectangular loop increases as it moves into the...
3.1K
Relative Motion Analysis using Rotating Axes
441
Consider a component AB undergoing a linear motion. Along with a linear motion, point B also rotates around point A. To comprehend this complex movement, position vectors for both points A and B are established using a stationary reference frame.
However, to express the relative position of point B relative to point A, an additional frame of reference, denoted as x'y', is necessary. This additional frame not only translates but also rotates relative to the fixed frame, making it...
However, to express the relative position of point B relative to point A, an additional frame of reference, denoted as x'y', is necessary. This additional frame not only translates but also rotates relative to the fixed frame, making it...
441
Equilibrium and Balance
4.3K
The inner ear assumes dual functionalities of auditory perception and equilibrium maintenance. The vestibule is the organ responsible for balance. This organ contains mechanoreceptors, specifically hair cells, endowed with stereocilia, which aid in deciphering information regarding the position and motion of our heads. Two intrinsic components, the utricle and saccule, help perceive head position, while the semicircular canals track head movement. Neurological messages initiated in the...
4.3K


