Human-Centric Relation Segmentation: Dataset and Solution.

Area of Science:

• Computer Vision
• Robotics
• Artificial Intelligence

Background:

• Existing vision-language models struggle with fine-grained details in human-object interactions, limiting robotic capabilities.
• Precisely identifying human body parts involved in interactions is crucial for tasks like object retrieval.

Purpose of the Study:

• Introduce Human-Centric Relation Segmentation (HRS) for detailed understanding of human-object interactions.
• Develop a novel framework for pixel-level segmentation of entities and identification of relation-specific human parts.

Main Methods:

• Propose the Simultaneous Matching and Segmentation (SMS) framework with parallel branches for entity segmentation, subject-object matching, and human parsing.
• Utilize dynamically-generated conditional convolutions for entity mask prediction and displacement estimation for relation linking.
• Collect and annotate the new Person In Context (PIC) dataset with 17,122 images, 141 object categories, 23 relation categories, and 25 human part categories.

Main Results:

• The SMS framework achieves state-of-the-art performance on the PIC and V-COCO datasets for the HRS task.
• Demonstrates high inference speed (36 FPS) and significantly outperforms existing methods, including m-KERN, with lower computational cost.
• Provides precise segmentation masks for objects and identifies relation-correlated human parts, enabling robots to perform complex grasping tasks.

Conclusions:

• Human-Centric Relation Segmentation (HRS) effectively addresses the challenge of fine-grained detail in human-object interaction understanding.
• The proposed SMS framework offers an efficient and accurate solution for robotic vision systems requiring detailed spatial and relational context.
• The new PIC dataset and SMS framework advance the development of more capable and context-aware robots.