An Underwater Human-Robot Interaction Using a Visual-Textual Model for Autonomous Underwater Vehicles

Area of Science:

• Robotics
• Computer Vision
• Marine Technology

Background:

• Human-robot interaction in marine environments is challenging.
• Underwater gesture recognition is difficult for autonomous underwater vehicles (AUVs) due to optical distortions.
• Existing methods often ignore valuable textual information.

Purpose of the Study:

• To develop an improved method for underwater hand gesture recognition.
• To enhance communication between divers and AUVs.
• To leverage multimodal information for more robust gesture recognition.

Main Methods:

• Proposed a novel visual-textual model for underwater hand gesture recognition (VT-UHGR).
• Encoded diver images as visual features and gesture categories as textual features.
• Utilized multimodal interactions to generate combined visual-textual features.
• Employed image-text matching for AUV learning and inference.

Main Results:

• The VT-UHGR model demonstrated superior performance compared to purely visual methods.
• Achieved better results on the CADDY dataset.
• Validated the effectiveness of incorporating textual patterns in underwater gesture recognition.

Conclusions:

• Integrating textual information significantly enhances underwater gesture recognition.
• The proposed visual-textual approach offers a more effective solution for diver-AUV communication.
• This method shows promise for improving underwater human-robot interaction.