LLaVA-Pose:关键点集成指令调整人类姿势和行动理解
Dewen Zhang1, Tahir Hussain1, Wangpeng An2
1Department of Informatics, Graduate School of Informatics and Engineering, The University of Electro-Communications, Tokyo 182-8585, Japan.
Sensors (Basel, Switzerland)
|August 28, 2025
概括
这项研究引入了关键点集成的数据,以改进视觉语言模型 (VLM) 来理解人类的姿势和行为. 通过精细调整这一专用数据集, 显著提高了以人为中心任务的VLM性能.
科学领域:
- 计算机视觉
- 人工智能
- 多模式学习
背景情况:
- 目前的视觉语言模型 (VLM) 在一般视觉任务中表现出色,
- 这种局限性源于缺乏以人为中心的视觉理解的专业指导数据.
研究的目的:
- 开发一种用于生成特殊视觉语言数据的方法,将人类关键点与传统视觉特征整合起来.
- 创建一个全面的数据集,以微调以人为本的任务,包括对话,详细描述和复杂的推理.
- 建立一个衡量人类姿势和行动表现的基准.
主要方法:
- 整合人类关键点数据与现有的视觉特征,如标题和边界框.
- 建立了200,328个样本的数据集,专注于以人为中心的任务.
- 建立了扩展人类姿势和行动理解基准 (E-HPAUB).
- 使用生成的数据集微调了LLaVA-1.5-7B模型以创建LLaVA-Pose模型.
主要成果:
- 在LLaVA-Pose模型中,对E-HPAUB基准进行了显著的改进.
- 与基线LLaVA-1. 5-7B模型相比,整体性能提高了33. 2%.
- 验证了关键点集成数据的有效性,以提高以人为中心的视觉理解.
结论:
- 关键点集成的数据对于理解复杂的人类姿势和行动的VLM至关重要.
- 拟议的方法和数据集有效地提高了以人为中心的视觉任务的多式模式能力.
相关概念视频
Absolute Motion Analysis- General Plane Motion
271
Visualize a drone, with its propellers spinning rapidly, hovering mid-air. The fascinating movements and operations of this drone can be comprehended by applying the principle of general plane motion.
As the drone's propellers rotate, an upward force is generated that counteracts the force of gravity, enabling the drone to lift off from the ground. This initial movement of the drone is along a straight path, representing a form of translational motion. In this phase, every point on the...
As the drone's propellers rotate, an upward force is generated that counteracts the force of gravity, enabling the drone to lift off from the ground. This initial movement of the drone is along a straight path, representing a form of translational motion. In this phase, every point on the...
271
Fixed Action Patterns
16.5K
A fixed action pattern (FAP) is a specific, hard-wired sequence of behaviors that occurs in response to an external stimulus, called a sign stimulus. The behavior is “fixed” because it is essentially unchangeable—proceeding similarly across individuals of a species every time it occurs.
16.5K
Muscle Coordination and Action
2.0K
Muscle coordination is a complex and finely tuned process essential for smooth and purposeful movements like flexion, extension, adduction, abduction, and rotation. The human body orchestrates the actions of various muscles working in concert, each with a specific role. Four functional types describe how muscles work together: agonist, antagonist, synergist, and fixator.
Agonists
Agonist muscles, often called prime movers, are the primary muscles responsible for producing a specific movement....
Agonists
Agonist muscles, often called prime movers, are the primary muscles responsible for producing a specific movement....
2.0K
Relative Motion Analysis using Rotating Axes-Problem Solving
448
Consider a crane whose telescopic boom rotates with an angular velocity of 0.04 rad/s and angular acceleration of 0.02 rad/s2. Along with the rotation, the boom also extends linearly with a uniform speed of 5 m/s. The extension of the boom is measured at point D, which is measured with respect to the fixed point C on the other end of the boom. For the given instant, the distance between points C and D is 60 meters.
Here, in order to determine the magnitude of velocity and acceleration for point...
Here, in order to determine the magnitude of velocity and acceleration for point...
448
Planar Rigid-Body Motion
545
Understanding the movement of a rigid body in planar motion involves recognizing that every particle within this body is traversing a path that maintains a consistent distance from a specific plane. This concept is fundamental in the study of physics and mechanical engineering, and it allows us to comprehend better how objects move in space.
Planar motion is typically divided into three distinct categories. The first is rectilinear translation, demonstrated by a subway train that moves along...
Planar motion is typically divided into three distinct categories. The first is rectilinear translation, demonstrated by a subway train that moves along...
545
Kinematic Equations: Problem Solving
13.6K
When analyzing one-dimensional motion with constant acceleration, the problem-solving strategy involves identifying the known quantities and choosing the appropriate kinematic equations to solve for the unknowns. Either one or two kinematic equations are needed to solve for the unknowns, depending on the known and unknown quantities. Generally, the number of equations required is the same as the number of unknown quantities in the given example. Two-body pursuit problems always require two...
13.6K


