Multimodal optimal matching and augmentation method for small sample gesture recognition

In human-computer interaction, gesture recognition based on physiological signals offers advantages such as a more natural and fast interaction mode and less constrained by the environment than visual-based. Surface electromyography-based gesture recognition has significantly progressed. However, since individuals have physical differences, researchers must collect data multiple times from each user to train the deep learning model. This data acquisition process can be particularly burdensome for non-healthy users. Researchers are currently exploring transfer learning and data augmentation techniques to enhance the accuracy of small-sample gesture recognition models. However, challenges persist, such as negative transfer and limited diversity in training samples, leading to suboptimal recognition performance. Therefore, We introduce motion information into sEMG-based recognition and propose a multimodal optimal matching and augmentation method for small sample gesture recognition, achieving efficient gesture recognition with only one acquisition per gesture. Firstly, this method utilizes the optimal matching signal selection module to select the most similar signals from the existing data to the new user as the training set, reducing inter-domain differences. Secondly, the similarity calculation augmentation module enhances the diversity of the training set. Finally, the Modal-type embedding enhances the information interaction between each mode signal. We evaluated the effectiveness on Self-collected Stroke Patient, the Ninapro DB1 dataset and the Ninapro DB5 dataset and achieved accuracies of 93.69%, 91.65% and 98.56%, respectively. These results demonstrate that the method achieved performance comparable to traditional recognition models while significantly reducing the collected data.