Channel-Wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition

Graph convolutional networks (GCNs) have been widely used and have achieved remarkable results in skeleton-based action recognition. In this paper, we propose a channel-wise topology refinement-graph convolution (GC) to dynamically learn different topologies and effectively aggregate joint features in different channels for skeleton-based action recognition. Channel-wise topologies are modeled by learning a shared topology as a generic prior for all the channels and refining the topology using channel-specific correlations between joints. Our refinement method introduces very few extra parameters and significantly reduces the difficulty in modeling channel-wise topologies. Furthermore, we reformulate graph convolutions (GCs) into a unified form, and theoretically show that the channel-wise topology refinement-GC relaxes strict constraints of GCs and then has stronger representation capability. In order to model long-range joint dependencies and dynamically adjust channel-wise feature weights, we propose a channel-wise topology refinement squeeze-excitation transformer. Global information is aggregated in the temporal and spatial dimensions to capture correlations between distant joints. The channel-attention mechanism extracts channel-level statistics using global average pooling. The channel weights are generated using a fully connected layer to reinforce action-adaptive feature channels. Experimental results show the effectiveness of the proposed channel-wise topology refinement-GC and channel-wise topology refinement squeeze-excitation transformer on the NTU RGB+D, NTU RGB+D120, and NW-UCLA datasets.