Point Cloud Hand-Object Segmentation Using Multimodal Imaging with Thermal and Color Data for Safe Robotic Object...

Area of Science:

• Robotics and Artificial Intelligence
• Computer Vision
• Sensor Fusion

Background:

• Safe human-robot object handover requires precise segmentation of hands and objects.
• Integrating multimodal 3D data (point cloud, RGB, thermal) presents calibration and alignment challenges.
• Existing methods lack robust performance in complex, real-world scenarios.

Purpose of the Study:

• To develop and validate a multimodal sensor system for accurate hand-object segmentation.
• To investigate the effectiveness of different data modalities for improving segmentation performance.
• To enable safer and more reliable human-robot interaction during object transfer.

Main Methods:

• A novel calibration target with an active light source was designed for simultaneous multi-camera capture (RGB, thermal, NIR).
• Neural networks (PointNet, PointNet++, RandLA-Net) were trained on multimodal 3D datasets (XYZ, XYZ-T, XYZ-RGB, XYZ-RGB-T).
• Performance was evaluated using Intersection over Union (IoU) metrics for hand-object segmentation.

Main Results:

• The XYZ-RGB-T data mode achieved the highest mean IoU of 82.8% using RandLA-Net.
• Segmentation performance significantly improved with the integration of RGB and thermal data compared to single modalities.
• The hand segmentation class alone reached an IoU of 92.6%.

Conclusions:

• Multimodal 3D data, particularly the combination of XYZ, RGB, and thermal information, is crucial for accurate hand-object segmentation.
• The proposed sensor system and calibration method effectively address multimodal data integration challenges.
• This approach significantly advances the capabilities for safe and precise human-robot object handover.