Framework for Armature-Based 3D Shape Reconstruction of Sensorized Soft Robots in eXtended Reality

Area of Science:

• Robotics
• Computer Vision
• Human-Computer Interaction

Background:

• Soft robots excel in unpredictable environments but require operator supervision for complex tasks.
• Direct observation of soft robots is often hindered by their operating environment.
• Interpreting complex 3D configurations of soft robots is challenging with traditional visualization.

Purpose of the Study:

• To present an open-source framework for real-time 3D reconstruction of soft robots.
• To enable intuitive teleoperation and control in challenging environments.
• To facilitate future research in remote control of soft robots.

Main Methods:

• Developed an open-source framework with a Robot Operating System (ROS) backbone.
• Integrated proprioceptive sensor data for 3D reconstruction.
• Utilized eXtended Reality (XR), specifically Augmented Reality (AR) with Microsoft Hololens, for visualization.
• Evaluated performance metrics like speed and scalability based on system parameters (mesh density, armature complexity).

Main Results:

• Achieved real-time 3D reconstruction of soft robots in XR at up to 60 FPS.
• Demonstrated successful application in Augmented Reality for intuitive teleoperation.
• Analyzed the impact of mesh density and armature complexity on reconstruction performance.

Conclusions:

• The developed framework provides a platform for real-time remote control of soft robots.
• Enhances operator supervision and task execution in environments that impede direct observation.
• Facilitates intuitive teleoperation through real-time 3D visualization in XR.