LLaVA-Pose: Keypoint-Integrated Instruction Tuning for Human Pose and Action Understanding.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Multimodal Learning

Background:

• Current vision-language models (VLMs) excel at general visual tasks but struggle with complex human pose and action recognition.
• This limitation stems from a lack of specialized instruction-following data for human-centric visual understanding.

Purpose of the Study:

• To develop a method for generating specialized vision-language data integrating human keypoints with traditional visual features.
• To create a comprehensive dataset for fine-tuning VLMs on human-centric tasks, including conversation, detailed description, and complex reasoning.
• To establish a benchmark for evaluating model performance in human pose and action understanding.

Main Methods:

• Integrated human keypoint data with existing visual features like captions and bounding boxes.
• Constructed a dataset of 200,328 samples focused on human-centric tasks.
• Established the Extended Human Pose and Action Understanding Benchmark (E-HPAUB).
• Fine-tuned the LLaVA-1.5-7B model using the generated dataset to create the LLaVA-Pose model.

Main Results:

• The LLaVA-Pose model demonstrated significant improvements on the E-HPAUB benchmark.
• Achieved an overall performance increase of 33.2% compared to the baseline LLaVA-1.5-7B model.
• Validated the effectiveness of keypoint-integrated data for enhancing human-centric visual understanding.

Conclusions:

• Keypoint-integrated data is crucial for advancing VLMs in understanding complex human poses and actions.
• The proposed method and dataset effectively improve multimodal model capabilities for human-centric visual tasks.