HANDS: a multimodal dataset for modeling toward human grasp intent inference in prosthetic hands

Area of Science:

• Robotics and Human-Computer Interaction
• Biomedical Engineering
• Computer Vision

Background:

• Upper limb and hand functionality is vital for daily activities, and amputation leads to significant loss.
• Advanced prosthetic hands require improved intent inference from multimodal sensor data for better control.
• Integrating environmental and human physiological/behavioral sensors is key for prosthetic perception.

Purpose of the Study:

• To present a novel dataset for estimating human intent for prosthetic hand control using hand-view imagery.
• To enable computer vision methods to interpret visual evidence from cameras integrated into prosthetic hands.
• To facilitate the fusion of environmental and human sensor data for prosthetic hand motion planning.

Main Methods:

• Captured paired eye-view and hand-view images of objects in various orientations.
• Recorded synchronized video, electromyography (EMG), and inertial measurement unit (IMU) data during grasp, lift, and put-down trials.
• Trained a convolutional neural network (CNN) on hand-view images to predict human-assigned grasp preferences.

Main Results:

• Demonstrated the utility of hand-view images for predicting human grasp intent.
• Successfully trained a CNN to interpret visual cues from the prosthetic hand's perspective.
• The dataset facilitates research into fusing multimodal sensor data for enhanced prosthetic control.

Conclusions:

• Hand-view imagery is a valuable source for inferring human intent in prosthetic hand applications.
• Computer vision applied to prosthetic-mounted cameras can significantly aid intent recognition.
• The developed dataset supports the advancement of intelligent prosthetic systems through multimodal data fusion.