Robust Human-to-Robot Handover System Under Adverse Lighting | JoVE Visualize

Area of Science:

Robotics
Computer Vision
Human-Robot Interaction

Background:

Human-to-robot (H2R) handovers are crucial for collaborative tasks but are hindered by environmental complexities affecting robot perception.
Traditional RGB-based perception struggles in adverse lighting (glare, darkness) and unstructured settings with fine-grained details like hand poses and object geometries.
Existing semantic segmentation and grasp generation methods lack generalization in these challenging H2R handover scenarios.

Purpose of the Study:

To develop a robust H2R handover system capable of reliable perception under varying lighting conditions.
To enhance 3D semantic segmentation and grasp generation for complex H2R handover environments.
To improve the overall success rate and generalization capabilities of H2R handover systems.

Main Methods:

A dual-path perception pipeline fusing stereo RGB-D and Time-of-Flight (ToF) camera data, switching to ToF in adverse lighting.
Augmented Point Transformer v3 (PTv3) with T-Net and self-attention for real-time 3D semantic segmentation of objects and human hands.
Extended GraspNet with a grasp selection module specifically optimized for H2R handover requirements.

Main Results:

Achieved 84.4% mIoU in semantic segmentation, outperforming Swin3D-L by 3.26 percentage points with 3.2x faster inference.
Demonstrated an 18.4 percentage point improvement in real-world handover success rate across various objects, postures, and angles.
Attained 82.7% success in adverse lighting and significantly outperformed a state-of-the-art multimodal system (75.0% vs. 15.0%) under identical conditions.

Conclusions:

The proposed dual-path perception system significantly enhances H2R handover robustness and reliability in challenging environments.
The augmented PTv3 and GraspNet extensions provide effective solutions for complex 3D semantic segmentation and grasp generation.
The system demonstrates strong generalization and robustness, paving the way for safer and more efficient human-robot collaboration.