Block-term tensor neural networks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Deep neural networks (DNNs) excel in various applications but suffer from high parameter counts, hindering efficient training and deployment on resource-constrained devices.
• The computational demands of DNNs limit their accessibility and scalability in real-world applications.

Purpose of the Study:

• To address the parameter inefficiency of DNNs by proposing a novel layer structure.
• To reduce the computational and memory footprint of DNNs without compromising their representational power.

Main Methods:

• Exploration of weight matrix correlations within DNNs.
• Approximation of weight matrices using low-rank block-term tensors.
• Introduction of block-term tensor layers (BT-layers) adaptable to Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs).
• Reshaping of inputs and outputs in BT-layers into low-dimensional, high-order tensors.

Main Results:

• BT-layers demonstrate significant compression ratios for DNN parameters.
• The proposed method preserves or enhances the representation power of original DNN models.
• Experiments confirm the effectiveness of BT-layers in both CNNs and RNNs.

Conclusions:

• Block-term tensor layers offer an effective strategy for compressing DNNs.
• BT-layers provide a viable solution for deploying deep learning models on devices with limited resources.
• This approach facilitates more efficient and scalable deep learning applications.