Compact Neural Architecture Designs by Tensor Representations

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Traditional neural networks often flatten higher-order tensor data, losing valuable multi-dimensional structural information.
• Existing methods for handling complex data structures in neural networks can be computationally expensive and less efficient.

Purpose of the Study:

• To introduce a novel framework of tensorial neural networks (TNNs) that naturally handle higher-order tensor data.
• To demonstrate the advantages of TNNs in preserving data structure, reducing model parameters, and interpreting compact network designs.

Main Methods:

• Developed a framework extending linear layers to multilinear operations on higher-order tensors.
• Derived backpropagation rules for TNNs using generalized tensor algebra.
• Applied knowledge distillation for training TNNs from pre-trained models or from scratch.

Main Results:

• TNNs preserve the multi-dimensional structure of higher-order data by avoiding flattening.
• Compressing pre-trained networks into TNNs yields models with similar expressive power but significantly fewer parameters.
• Experiments on VGG, ResNet, and Wide-ResNet show TNNs outperform state-of-the-art low-rank methods.

Conclusions:

• Tensorial neural networks offer a powerful and efficient approach for deep learning with higher-order data.
• TNNs provide a method for model compression and interpretation of advanced network architectures.
• The proposed framework demonstrates superior performance across various backbone networks and datasets.