Robust Human-to-Robot Handover System Under Adverse Lighting

Area of Science:

• Robotics
• Computer Vision
• Human-Robot Interaction

Background:

• Human-to-robot (H2R) handovers are crucial for collaborative tasks but are hindered by environmental complexities affecting robot perception.
• Traditional RGB-based perception struggles in adverse lighting (glare, darkness) and unstructured settings with fine-grained details like hand poses and object geometries.
• Existing semantic segmentation and grasp generation methods lack generalization in these challenging H2R handover scenarios.

Purpose of the Study:

• To develop a robust H2R handover system capable of reliable perception under varying lighting conditions.
• To enhance 3D semantic segmentation and grasp generation for complex H2R handover environments.
• To improve the overall success rate and generalization capabilities of H2R handover systems.

Main Methods:

• A dual-path perception pipeline fusing stereo RGB-D and Time-of-Flight (ToF) camera data, switching to ToF in adverse lighting.
• Augmented Point Transformer v3 (PTv3) with T-Net and self-attention for real-time 3D semantic segmentation of objects and human hands.
• Extended GraspNet with a grasp selection module specifically optimized for H2R handover requirements.

Main Results:

• Achieved 84.4% mIoU in semantic segmentation, outperforming Swin3D-L by 3.26 percentage points with 3.2x faster inference.
• Demonstrated an 18.4 percentage point improvement in real-world handover success rate across various objects, postures, and angles.
• Attained 82.7% success in adverse lighting and significantly outperformed a state-of-the-art multimodal system (75.0% vs. 15.0%) under identical conditions.

Conclusions:

• The proposed dual-path perception system significantly enhances H2R handover robustness and reliability in challenging environments.
• The augmented PTv3 and GraspNet extensions provide effective solutions for complex 3D semantic segmentation and grasp generation.
• The system demonstrates strong generalization and robustness, paving the way for safer and more efficient human-robot collaboration.