Visual Scene Understanding for Enhanced EMG Gesture Recognition

This work presents a multimodal approach combining electromyography (EMG) and computer vision (CV) for robust real-time gesture recognition in a real-world setting. A context-aware framework is proposed for myoelectric prosthesis control, wherein EMG hand gesture recognition is augmented by the visual detection of objects of interest in a scene, effectively mitigating risks of false movements. By supporting EMG gesture predictions produced by a Siamese deep convolution neural network (SDCNN) with context derived from object detection using a tailored YOLO computer vision model, the system prevents false detection during gesture onset and during static gesture maintenance. In a pilot experiment, this multimodal sensor fusion is shown to effectively augment human volitional control by enhancing both the robustness of the gesture control interface and the user's ability to maintain full command over it.