Deep Manifold Learning Combined With Convolutional Neural Networks for Action Recognition

Learning deep representations have been applied in action recognition widely. However, there have been a few investigations on how to utilize the structural manifold information among different action videos to enhance the recognition accuracy and efficiency. In this paper, we propose to incorporate the manifold of training samples into deep learning, which is defined as deep manifold learning (DML). The proposed DML framework can be adapted to most existing deep networks to learn more discriminative features for action recognition. When applied to a convolutional neural network, DML embeds the previous convolutional layer's manifold into the next convolutional layer; thus, the discriminative capacity of the next layer can be promoted. We also apply the DML on a restricted Boltzmann machine, which can alleviate the overfitting problem. Experimental results on four standard action databases (i.e., UCF101, HMDB51, KTH, and UCF sports) show that the proposed method outperforms the state-of-the-art methods.