Cross-Viewpoint Semantic Mapping: Integrating Human and Robot Perspectives for Improved 3D Semantic Reconstruction

Area of Science:

• Robotics
• Computer Vision
• Artificial Intelligence

Background:

• Allocentric semantic 3D maps are crucial for human-machine interaction, enabling egocentric viewpoint derivation.
• Discrepancies in class labels and map interpretations arise from differing human and robot perspectives, especially with small robots.
• Existing 3D semantic reconstruction pipelines struggle with viewpoint variations.

Purpose of the Study:

• To extend 3D semantic reconstruction pipelines with cross-viewpoint semantic matching for human-robot collaboration.
• To develop methods for acquiring semantic labels from unusual, low-angle robot viewpoints.
• To enable small robots to generate high-quality semantic maps for human partners.

Main Methods:

• Extending a real-time 3D semantic reconstruction pipeline with semantic matching across human and robot viewpoints.
• Utilizing deep recognition networks and proposing novel approaches for acquiring semantic labels from non-standard perspectives.
• Adapting human-perspective semantic reconstructions to robot viewpoints using superpixel segmentation and environmental geometry.

Main Results:

• Achieved high-quality semantic segmentation from the robot's perspective, comparable to human viewpoints.
• Improved deep network recognition performance for low viewpoints by exploiting gained semantic information.
• Demonstrated the capability of a small robot to generate high-quality semantic maps autonomously.

Conclusions:

• The proposed approach effectively bridges the semantic gap between human and robot viewpoints in 3D mapping.
• Enables small robots to create accurate and useful semantic maps, fostering better human-robot interaction.
• Real-time performance facilitates interactive applications in robotics and augmented reality.