空间时间图曼巴用于音乐引导舞蹈视频合成
概括
这项研究介绍了STG-Mamba,这是一个新的AI模型,用于从音乐中生成舞蹈视频. 它有效地将音乐转化为人类骨运动,然后转化为现实的舞蹈视频,优于现有的方法.
科学领域:
- 计算机视觉 计算机视觉
- 人工智能的人工智能
- 机器学习 机器学习
背景情况:
- 以音乐为指导的舞蹈视频合成是一项具有挑战性的任务,需要准确地将听觉输入转化为复杂的人类运动.
- 现有的方法往往难以捕捉舞蹈运动中固有的复杂的空间和时间动态.
研究的目的:
- 为音乐引导的舞蹈视频合成提出一个新的时空图马巴 (STG-Mamba) 模型.
- 开发有效的模块,用于音乐到骨架和骨架到视频的翻译.
主要方法:
- 引入了一个时空图马巴 (STGM) 块来建模音乐到骨的翻译,捕捉跨空间和时间的联合依赖.
- 开发了一个自主监督的规范化网络,用于骨架到视频的翻译,并结合了条件图像.
- 创建并使用了54,944个视频剪辑的新数据集,用于骨到视频的翻译.
主要成果:
- 与现有的方法相比,STG-Mamba模型在以音乐为指导的舞蹈视频合成中表现出卓越的性能.
- 该STGM块有效地捕获了骨序列中的复杂的时空依赖.
- 自主监督的规范化网络能够准确地将骨翻译成真实的舞蹈视频.
结论:
- STG-Mamba提供了从音乐到人工智能驱动的舞蹈视频生成的重大进步.
- 拟议的STGM封锁和规范化网络是这项任务的有效组成部分.
- 开发的数据集有助于进一步研究骨到视频翻译.
相关概念视频
Relative Motion Analysis using Rotating Axes-Problem Solving
451
Consider a crane whose telescopic boom rotates with an angular velocity of 0.04 rad/s and angular acceleration of 0.02 rad/s2. Along with the rotation, the boom also extends linearly with a uniform speed of 5 m/s. The extension of the boom is measured at point D, which is measured with respect to the fixed point C on the other end of the boom. For the given instant, the distance between points C and D is 60 meters.
Here, in order to determine the magnitude of velocity and acceleration for point...
Here, in order to determine the magnitude of velocity and acceleration for point...
451
Relative Motion Analysis using Rotating Axes
539
Consider a component AB undergoing a linear motion. Along with a linear motion, point B also rotates around point A. To comprehend this complex movement, position vectors for both points A and B are established using a stationary reference frame.
However, to express the relative position of point B relative to point A, an additional frame of reference, denoted as x'y', is necessary. This additional frame not only translates but also rotates relative to the fixed frame, making it...
However, to express the relative position of point B relative to point A, an additional frame of reference, denoted as x'y', is necessary. This additional frame not only translates but also rotates relative to the fixed frame, making it...
539
Velocity and Position by Graphical Method
8.2K
Velocity and position can be calculated from the known function of acceleration as a function of time. The total area under the acceleration-time graph and the velocity-time graph gives the change in velocity and position, respectively. In the case of an airplane, its acceleration is tracked using the inertial navigation system. The pilot provides the input of the airplane's initial position and velocity before takeoff. The inertial navigation system then uses the acceleration data to...
8.2K
Absolute Motion Analysis- General Plane Motion
273
Visualize a drone, with its propellers spinning rapidly, hovering mid-air. The fascinating movements and operations of this drone can be comprehended by applying the principle of general plane motion.
As the drone's propellers rotate, an upward force is generated that counteracts the force of gravity, enabling the drone to lift off from the ground. This initial movement of the drone is along a straight path, representing a form of translational motion. In this phase, every point on the...
As the drone's propellers rotate, an upward force is generated that counteracts the force of gravity, enabling the drone to lift off from the ground. This initial movement of the drone is along a straight path, representing a form of translational motion. In this phase, every point on the...
273
Relative Motion Analysis - Velocity
439
A stroke engine has a slider-crank mechanism that converts rotational motion from the crank into linear motion of the slider or vice versa. This mechanism consists of three main parts: the crank, the connecting rod, and the slider.
When an external force is exerted, it sets the crank into a rotational movement. This, in turn, instigates the motion of the connecting rod, leading to what is referred to as a general plane motion. This process involves two key points - point A on the connecting rod...
When an external force is exerted, it sets the crank into a rotational movement. This, in turn, instigates the motion of the connecting rod, leading to what is referred to as a general plane motion. This process involves two key points - point A on the connecting rod...
439
Time-Series Graph
4.5K
A time-series graph is a line graph with repeated measurements taken at successive intervals of time. It is also called a time series chart. To construct a time-series graph, one must look at both pieces of a paired data set. The horizontal axis is used to plot the time increments, and the vertical axis is used to plot the values of the variable that one is measuring. By using the axes in this way, each point on the graph will correspond to time and a measured quantity. The points on the graph...
4.5K


