运动LLM:从人类运动和视频了解人类行为
IEEE transactions on pattern analysis and machine intelligence
|October 31, 2025
概括
本研究介绍了MotionLLM,这是一种使用视频和运动数据来理解人类行为的框架. 它通过共同建模多模式输入来增强时空理解和推理.
科学领域:
- 计算机视觉 计算机视觉
- 人工智能的人工智能
- 人类行为分析 人类行为分析
背景情况:
- 当前的大型语言模型 (LLM) 往往专注于单个模式 (视频或运动) 来理解人类行为.
- 有效的人类行为分析需要整合视觉 (视频) 和动态 (运动) 数据的信息,以捕捉详细的身体动态.
研究的目的:
- 开发一个新的框架,MotionLLM,以全面理解人类行为,包括标题和推理.
- 提出一个统一的培训策略,利用视频-文本和动作-文本对的互补数据.
- 引入一个新的数据集 (MoVid) 和基准 (MoVid-Bench) 来评估多模式的人类行为理解.
主要方法:
- 开发了MotionLLM,这是一个整合视频和运动数据用于人类行为分析的框架.
- 实施了一种统一的视频运动训练策略,使用粗略的视频文本和细粒度的运动文本数据.
- 收集了MoVid数据集并创建了MoVid-Bench以进行可靠的评估.
主要成果:
- MotionLLM在人类动作字幕中表现出卓越的性能.
- 该框架显示了空间时间理解能力的增强.
- 运动LLM在人类行为理解任务中表现出改进的推理能力.
结论:
- 联合建模视频和运动数据对于了解微妙的人类行为至关重要.
- MotionLLM提供了一种有效的方法,用于多模式的人类行为分析.
- MoVid数据集和MoVid-Bench为未来的研究提供了宝贵的资源.
相关概念视频
Absolute Motion Analysis- General Plane Motion
522
Visualize a drone, with its propellers spinning rapidly, hovering mid-air. The fascinating movements and operations of this drone can be comprehended by applying the principle of general plane motion.
As the drone's propellers rotate, an upward force is generated that counteracts the force of gravity, enabling the drone to lift off from the ground. This initial movement of the drone is along a straight path, representing a form of translational motion. In this phase, every point on the...
As the drone's propellers rotate, an upward force is generated that counteracts the force of gravity, enabling the drone to lift off from the ground. This initial movement of the drone is along a straight path, representing a form of translational motion. In this phase, every point on the...
522
Relative Motion Analysis - Velocity
679
A stroke engine has a slider-crank mechanism that converts rotational motion from the crank into linear motion of the slider or vice versa. This mechanism consists of three main parts: the crank, the connecting rod, and the slider.
When an external force is exerted, it sets the crank into a rotational movement. This, in turn, instigates the motion of the connecting rod, leading to what is referred to as a general plane motion. This process involves two key points - point A on the connecting rod...
When an external force is exerted, it sets the crank into a rotational movement. This, in turn, instigates the motion of the connecting rod, leading to what is referred to as a general plane motion. This process involves two key points - point A on the connecting rod...
679
Relative Motion Analysis using Rotating Axes
870
Consider a component AB undergoing a linear motion. Along with a linear motion, point B also rotates around point A. To comprehend this complex movement, position vectors for both points A and B are established using a stationary reference frame.
However, to express the relative position of point B relative to point A, an additional frame of reference, denoted as x'y', is necessary. This additional frame not only translates but also rotates relative to the fixed frame, making it...
However, to express the relative position of point B relative to point A, an additional frame of reference, denoted as x'y', is necessary. This additional frame not only translates but also rotates relative to the fixed frame, making it...
870


