Identifying the Nature of Grip Force Signals in EEG & fNIRS with Multi-Modal Graph Fusion Network

Brain-Computer interfaces can assist motor rehabilitation for people with severe paralysis by directly decoding their brain signals into movement intention and executing with external devices without passing the impaired neural pathways. It is crucial to restore natural and smooth daily movements, and continuous force control is one of the most important kinaesthetic functions. However, the complex continuous force decoding and limited relevant public datasets greatly challenge this field. How the brain coordinates the motor command or sensory feedback during the force control behaviour also remains to be discussed. This work investigated these questions through a novel experimental setup by isolating the motor intention and sensory feedback and combining both components flexibly for hand grip. We applied functional electrical stimulation to induce passive gripping and collected grip force with multi-modal brain signals. Significant neural pattern differences were found in EEG time-frequency representation by comparing the brain responses under different task conditions, including voluntary movement, motor imagery, and passive perception status. Additionally, we present a multi-modal graph fusion model fusing both EEG and fNIRS for continuous bimanual grip force decoding. These contributions are beneficial to developing neural interfaces for rehabilitation and assistive devices that involve force manipulation or operate in isometric schemes.