Visual-to-Tactile Cross-Modal Generation Using a Class-Conditional GAN with Multi-Scale Discriminator and Hybrid Loss

Area of Science:

• Computer Vision
• Haptics
• Machine Learning

Background:

• Visual texture perception is vital for haptic rendering and virtual reality.
• Translating visual texture data into tactile feedback presents significant challenges.

Purpose of the Study:

• To develop a class-conditional Generative Adversarial Network (cGAN) for cross-modal translation from texture images to vibrotactile spectrograms.
• To enhance texture class semantics in the translation process for improved tactile realism.

Main Methods:

• Utilized a pix2pix-adapted generator with Conditional Batch Normalization (CBN) and a DenseNet-201 label predictor.
• Employed a multi-scale discriminator architecture (derived from pix2pixHD) for optimal perceptual similarity.
• Implemented a hybrid loss function combining adversarial, L1, and feature matching losses.

Main Results:

• Achieved superior perceptual similarity using Learned Perceptual Image Patch Similarity (LPIPS) and Fréchet Inception Distance (FID) metrics.
• Demonstrated improved performance over existing models like pix2pix and pix2pixHD.
• GradCAM visualizations confirmed the effectiveness of class conditioning.

Conclusions:

• The proposed cGAN model effectively translates visual texture information into vibrotactile spectrograms.
• Generated spectrograms can be converted to tactile signals, enabling advanced haptic feedback and virtual material simulation.
• This work advances the field of cross-modal translation for immersive virtual experiences.