Simultaneous Material Segmentation and 3D Reconstruction in Industrial Scenarios

Area of Science:

• Robotics
• Computer Vision
• Nuclear Engineering

Background:

• Accurate material recognition is crucial for nuclear waste decommissioning and segregation.
• Existing methods often require extensive labeled data, limiting their practical application.
• Robotic systems need robust perception for manipulation and navigation in hazardous environments.

Purpose of the Study:

• To develop a data-efficient transfer learning approach for boundary-aware material segmentation.
• To integrate this segmentation method with a Simultaneous Localization and Mapping (SLAM) system for 3D semantic mapping.
• To enhance robotic capabilities in nuclear waste management and disaster response scenarios.

Main Methods:

• Utilized transfer learning from a general computer vision dataset and weakly annotated material data.
• Employed a novel approach requiring limited pixel-wise annotations for material segmentation.
• Integrated the segmentation model with a SLAM system to generate a 3D global semantic map.

Main Results:

• Achieved quasi-real-time 3D semantic mapping with high-resolution images on the Materials in Context dataset (23 categories).
• Demonstrated successful verification in an industrial setting as part of the EU RoMaNs project.
• Presented promising qualitative results showcasing the system's effectiveness.

Conclusions:

• The proposed transfer learning method significantly improves data efficiency for material segmentation in robotic applications.
• Integration with SLAM enables the creation of detailed 3D semantic maps crucial for robot manipulation and navigation.
• The system shows strong potential for real-world deployment in nuclear decommissioning and disaster zones.