Hierarchical Transformer Fusion of Gaze Attention and Muscle Activity for Forearm Movement Estimation

Tracking forearm movement via measured physiological signals is crucial for understanding human motor control mechanism. Current methods mainly use muscle-derived signals to predict arm movements while often overlooking the potential role of gaze attention, which is important for hand-eye coordination and instant and continuous motion planning and execution. In this study, we explored the impact of gaze on motion tracking. A hierarchical transformer-based structure was developed to integrate gaze into muscle activity signals for recovering the joint trajectory. To collect the dataset, six subjects were recruited to perform arm motions broadly involved in daily activities; the measured signals from the muscle activity and gaze attention were used to train and evaluate the proposed method. A performance comparison was conducted between the models using solely muscle activity signals and both muscle and gaze information. The experimental results showed the important role of gaze information involved in motion prediction and the motor control mechanism. This research also gained insights on how to integrate gaze information into the muscle signals, which offers an alternative to bringing artificial intelligence to be engaged in the framework of motion tracking. Consequently, it is important for future designs of biomechanical sensors and wearable robotics systems.