Multimodal Deep Learning Model for Cylindrical Grasp Prediction Using Surface Electromyography and Contextual Data...

Area of Science:

• Biomedical Engineering
• Human-Machine Interaction
• Robotics

Background:

• Grasping is fundamental to daily activities and advanced human-machine systems.
• Current grasp prediction models often use limited, unimodal data sources.
• Developing sophisticated prosthetics and robotic hands requires accurate grasp type identification.

Purpose of the Study:

• To enhance cylindrical grasp prediction by integrating surface electromyography (EMG) signals with contextual data.
• To explore and compare different model architectures for multimodal grasp prediction.
• To determine the predictive power of task- and product-related contextual information.

Main Methods:

• Collected surface electromyography (EMG) signals and contextual data during object manipulation tasks.
• Developed three model architectures: EMG-based, context-based, and a hybrid multimodal model.
• Evaluated model performance using variables like object size, weight, and task height.

Main Results:

• Contextual information, particularly product-related features (object size and weight), demonstrated significant predictive power.
• The hybrid multimodal model integrating EMG and product context outperformed models using single data sources.
• Product context was more influential than task context (task height) in improving grasp prediction.

Conclusions:

• Integrating contextual data, especially product-related information, substantially improves EMG-based grasp prediction models.
• Multimodal approaches combining physiological signals with contextual data are crucial for advanced prosthetic and robotic hand control.
• Object properties are key determinants in predicting grasping strategies.