A multimodal multitask deep learning framework for vibrotactile feedback and sound rendering.

Area of Science:

• Human-Computer Interaction
• Deep Learning
• Signal Processing

Background:

• Previous research often treated vibrotactile feedback and sound generation separately.
• Synchronization and design complexity were key challenges in multimodal feedback systems.

Purpose of the Study:

• To introduce a novel multimodal multitask deep learning framework for integrated vibrotactile and sound rendering.
• To develop an end-to-end data-driven system for capturing and generating sensory feedback from tool-surface interactions.

Main Methods:

• Developed a comprehensive system capturing contact acceleration and sound data from textures.
• Introduced novel encoder-decoder networks with stacked transformers and convolutional layers.
• Applied a transformer-based, data-driven approach for modeling and rendering vibrotactile signals and sounds.

Main Results:

• The proposed framework demonstrated lower Root Mean Square (RMS) error compared to state-of-the-art models for both vibrotactile and sound data.
• Subjective evaluations confirmed the superiority of the proposed method over existing approaches.
• Achieved accurate synchronization and rendering of multimodal sensory feedback.

Conclusions:

• The novel multimodal multitask deep learning framework effectively models and generates synchronized vibrotactile feedback and sound.
• This approach overcomes previous limitations in addressing multiple sensory modalities simultaneously.
• Represents a significant advancement in data-driven approaches for realistic tool-surface interaction feedback.