LLaVA-ポーズ:人間のポーズと行動を理解するためのキーポイント統合のインストラクション
Dewen Zhang1, Tahir Hussain1, Wangpeng An2
1Department of Informatics, Graduate School of Informatics and Engineering, The University of Electro-Communications, Tokyo 182-8585, Japan.
Sensors (Basel, Switzerland)
|August 28, 2025
まとめ
この研究では,人間の姿勢と行動を理解するための視覚言語モデル (VLMs) を改善するために,キーポイント統合データを導入しています. この特殊なデータセットの微調整により,人間中心のタスクの VLM のパフォーマンスは大幅に向上します.
科学分野:
- コンピュータ・ビジョン
- 人工知能
- マルチモダルの学習
背景:
- 現在の視覚言語モデル (VLM) は 一般的な視覚的なタスクに優れていますが 複雑な人間の姿勢と行動認識に苦労しています
- この制限は,人間中心の視覚的理解のための特殊な指示に従うデータの欠如から生じる.
研究 の 目的:
- 人間のキーポイントと伝統的な視覚的特徴を統合した特殊な視覚言語データを生成する方法を開発する.
- 会話,詳細な説明,複雑な推論を含む人間中心的なタスクのVLMを微調整するための包括的なデータセットを作成します.
- 人間の姿勢と行動の理解におけるモデルのパフォーマンスを評価するための基準を確立する.
主な方法:
- キャプションや境界ボックスのような既存の視覚機能と統合された人間のキーポイントデータです.
- 人間中心のタスクに焦点を当てた200,328個のサンプルからなるデータセットを構築しました.
- 拡張人間姿勢と行動理解基準 (E-HPAUB) を確立しました.
- LLaVA-1.5-7Bモデルは,生成されたデータセットを使用して,LLaVA-Poseモデルを作成しました.
主要な成果:
- LLaVA-Poseモデルでは,E-HPAUBの基準値に大幅な改善が示されました.
- ベースラインのLLaVA-1. 5 - 7Bモデルと比較して33. 2%の全体的なパフォーマンスの向上を達成しました.
- 人間中心の視覚的理解を高めるためのキーポイント統合データの有効性を検証した.
結論:
- キーポイントに統合されたデータは,複雑な人間の姿勢と行動を理解するVLMを進めるために不可欠です.
- 提案された方法とデータセットは,人間中心の視覚的なタスクのためのマルチモダルのモデルの能力を効果的に改善します.
関連する概念動画
Absolute Motion Analysis- General Plane Motion
271
Visualize a drone, with its propellers spinning rapidly, hovering mid-air. The fascinating movements and operations of this drone can be comprehended by applying the principle of general plane motion.
As the drone's propellers rotate, an upward force is generated that counteracts the force of gravity, enabling the drone to lift off from the ground. This initial movement of the drone is along a straight path, representing a form of translational motion. In this phase, every point on the...
As the drone's propellers rotate, an upward force is generated that counteracts the force of gravity, enabling the drone to lift off from the ground. This initial movement of the drone is along a straight path, representing a form of translational motion. In this phase, every point on the...
271
Fixed Action Patterns
16.5K
A fixed action pattern (FAP) is a specific, hard-wired sequence of behaviors that occurs in response to an external stimulus, called a sign stimulus. The behavior is “fixed” because it is essentially unchangeable—proceeding similarly across individuals of a species every time it occurs.
16.5K
Muscle Coordination and Action
2.0K
Muscle coordination is a complex and finely tuned process essential for smooth and purposeful movements like flexion, extension, adduction, abduction, and rotation. The human body orchestrates the actions of various muscles working in concert, each with a specific role. Four functional types describe how muscles work together: agonist, antagonist, synergist, and fixator.
Agonists
Agonist muscles, often called prime movers, are the primary muscles responsible for producing a specific movement....
Agonists
Agonist muscles, often called prime movers, are the primary muscles responsible for producing a specific movement....
2.0K
Relative Motion Analysis using Rotating Axes-Problem Solving
448
Consider a crane whose telescopic boom rotates with an angular velocity of 0.04 rad/s and angular acceleration of 0.02 rad/s2. Along with the rotation, the boom also extends linearly with a uniform speed of 5 m/s. The extension of the boom is measured at point D, which is measured with respect to the fixed point C on the other end of the boom. For the given instant, the distance between points C and D is 60 meters.
Here, in order to determine the magnitude of velocity and acceleration for point...
Here, in order to determine the magnitude of velocity and acceleration for point...
448
Planar Rigid-Body Motion
545
Understanding the movement of a rigid body in planar motion involves recognizing that every particle within this body is traversing a path that maintains a consistent distance from a specific plane. This concept is fundamental in the study of physics and mechanical engineering, and it allows us to comprehend better how objects move in space.
Planar motion is typically divided into three distinct categories. The first is rectilinear translation, demonstrated by a subway train that moves along...
Planar motion is typically divided into three distinct categories. The first is rectilinear translation, demonstrated by a subway train that moves along...
545
Kinematic Equations: Problem Solving
13.6K
When analyzing one-dimensional motion with constant acceleration, the problem-solving strategy involves identifying the known quantities and choosing the appropriate kinematic equations to solve for the unknowns. Either one or two kinematic equations are needed to solve for the unknowns, depending on the known and unknown quantities. Generally, the number of equations required is the same as the number of unknown quantities in the given example. Two-body pursuit problems always require two...
13.6K


