Transfer of Learning from Vision to Touch: A Hybrid Deep Convolutional Neural Network for Visuo-Tactile 3D Object...

Area of Science:

• Robotics and Machine Intelligence
• Computer Vision
• Sensory Data Processing

Background:

• Transfer learning is widely used in machine intelligence, adapting pre-trained models for new tasks.
• Neuroscience suggests visual and tactile stimuli activate similar brain networks.
• This work explores transferring visual learning to tactile 3D object recognition.

Purpose of the Study:

• To investigate the effectiveness of transferring deep convolutional neural networks (CNNs) pre-trained on visual data for tactile 3D object recognition.
• To evaluate different pre-trained CNN architectures and tactile sensing technologies for this transfer learning task.
• To develop a hybrid model for combined visual and tactile object recognition.

Main Methods:

• Adapted five pre-trained CNN architectures for tactile datasets from five sensor types (BathTip, Gelsight, FSR, virtual FSR, Barrett robotic hand).
• Analyzed convolutional layer weight updates to assess visual-tactile feature similarity.
• Proposed a hybrid MobileNetV2-based architecture for integrated visual and tactile recognition.

Main Results:

• Confirmed the transferability of visual learning to tactile 3D model interpretation.
• Optical tactile sensors achieved higher classification rates than pressure-based sensors due to higher resolution.
• Efficient classification of tactile data was achieved by fine-tuning only a few layers of visually pre-trained CNNs.

Conclusions:

• Transfer learning from vision to touch is feasible for 3D object recognition.
• High-resolution optical tactile sensors are superior for leveraging visual features.
• A hybrid MobileNetV2 architecture achieved 100% visual and 77.63% tactile accuracy, suitable for mobile deployment.