A novel decoding method for motor imagery tasks with 4D data representation and 3D convolutional neural networks

Objective. Motor imagery electroencephalography (MI-EEG) produces one of the most commonly used biosignals in intelligent rehabilitation systems. The newly developed 3D convolutional neural network (3DCNN) is gaining increasing attention for its ability to recognize MI tasks. The key to successful identification of movement intention is dependent on whether the data representation can faithfully reflect the cortical activity induced by MI. However, the present data representation, which is often generated from partial source signals with time-frequency analysis, contains incomplete information. Therefore, it would be beneficial to explore a new type of data representation using raw spatiotemporal dipole information as well as the possible development of a matching 3DCNN.Approach.Based on EEG source imaging and 3DCNN, a novel decoding method for identifying MI tasks is proposed, called ESICNND. MI-EEG is mapped to the cerebral cortex by the standardized low resolution electromagnetic tomography algorithm, and the optimal sampling points of the dipoles are selected as the time of interest to best reveal the difference between any two MI tasks. Then, the initial subject coordinate system is converted to a magnetic resonance imaging coordinate system, followed by dipole interpolation and volume down-sampling; the resulting 3D dipole amplitude matrices are merged at the selected sampling points to obtain 4D dipole feature matrices (4DDFMs). These matrices are augmented by sliding window technology and input into a 3DCNN with a cascading architecture of three modules (3M3DCNN) to perform the extraction and classification of comprehensive features.Main results.Experiments are carried out on two public datasets; the average ten-fold CV classification accuracies reach 88.73% and 96.25%, respectively, and the statistical analysis demonstrates outstanding consistency and stability.Significance.The 4DDFMs reveals the variation of cortical activation in a 3D spatial cube with a temporal dimension and matches the 3M3DCNN well, making full use of the high-resolution spatiotemporal information from all dipoles.