Hulk: A Universal Knowledge Translator for Human-Centric Tasks.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Human-centric perception tasks like pedestrian detection and pose estimation are vital for applications such as the metaverse and sports analysis.
• Existing foundation models for human-centric tasks often lack 3D and vision-language capabilities and require task-specific fine-tuning, limiting their versatility.
• This necessitates the development of more generalized models that can handle a wider array of human-centric perception challenges.

Purpose of the Study:

• To introduce Hulk, the first multimodal human-centric generalist model.
• To enable a single model to address 2D vision, 3D vision, skeleton-based, and vision-language tasks without task-specific fine-tuning.
• To integrate knowledge across diverse human-centric tasks by treating them as modality translation problems.

Main Methods:

• Developed a novel approach by condensing various task-specific heads into two general heads: one for discrete representations (e.g., language) and one for continuous representations (e.g., coordinates).
• Enabled four distinct input and output modalities by stacking the outputs of these two general heads.
• Unified diverse human-centric tasks under a single framework, facilitating knowledge integration and cross-task learning.

Main Results:

• Hulk demonstrated superior performance across 12 benchmarks covering 8 human-centric tasks.
• Achieved state-of-the-art results in 11 out of the 12 evaluated benchmarks.
• Successfully handled 2D vision, 3D vision, skeleton-based action recognition, and vision-language tasks within a single, unified model.

Conclusions:

• Hulk represents a significant advancement in human-centric generalist models.
• The proposed method effectively integrates knowledge across a wide range of human-centric tasks through a unified representation.
• The model's ability to perform multiple tasks without fine-tuning opens new possibilities for efficient and versatile AI applications in human-centric domains.