Decoding imagined Chinese speech: a capsule neural network based on bidirectional knowledge transfer for hierarchical...

Objective.Speech imagination is one of the most important research directions in the field of brain-computer interfaces (BCIs). However, there is insufficient research on silent BCIs based on the Chinese stimulating materials. An experimental paradigm for Chinese speech imagery, which features a distinctive initial-and-final structure, is designed in this paper. According to the characteristics of vocalization and structure, the collected EEG data can be organized into a multi-level tree structure. Compared to conventional multi-label classification, our paper aims to study how to effectively utilize hierarchical structural information in the multi-granularity hierarchical classification tasks.Approach.We propose a hierarchical capsule network based on bidirectional knowledge transfer by using multi-band feature matrix, which is tailor-made for the phonological structure of Mandarin Chinese. The adoption of capsule network as the primary architecture is mainly due to the dynamic routing mechanism that can naturally model hierarchical relationships in the syllable hierarchy. In addition, we introduce the bidirectional knowledge transfer strategy to further improve the classical dynamic routing. Specifically, features from coarse-grained levels are added to fine-grained levels to fully utilize the dependency information between levels. In order to mitigate error propagation in the forward learning process, we also employ reverse knowledge transfer constrained via soft labels.Main results.The hierarchical classification results and ablation experiments both demonstrate the effectiveness of our proposed algorithm. The highest recognition rates for each layer reach 90.86%, 73.69%, and 69.45%, respectively.Significance.This article offers a novel perspective for decoding hierarchical Chinese silent BCI paradigms. Our study not only reveals the potential of linguistic domain knowledge in guiding neural network architectures for task-specific applications, but also provides a robust foundation for future individual phoneme classification.