Parameter-Efficient Deep Neural Networks With Bilinear Projections.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep neural networks (DNNs) often achieve higher accuracy with increased depth or width.
• However, larger DNNs lead to significant parameter redundancy, increasing computational and memory overhead.
• This limits the practical deployment of complex models, especially on resource-constrained devices.

Purpose of the Study:

• To address parameter redundancy in DNNs.
• To develop a parameter-efficient deep learning model suitable for mobile systems.
• To maintain or improve model accuracy while reducing computational and memory footprints.

Main Methods:

• Replaced conventional full projections in DNNs with bilinear projections (BPs).
• Analyzed the reduction in model space complexity from O(D^2) for full projections to O(2D) for BPs.
• Scaled up the mapping size by increasing output channels to counteract potential underfitting from structured projections.

Main Results:

• Bilinear projections significantly reduced model space complexity, achieving sublinear layer size.
• The proposed method maintained and even boosted model accuracy compared to conventional DNNs.
• Experiments demonstrated substantial reductions in model size and memory usage across four benchmark datasets.

Conclusions:

• Bilinear projections offer a parameter-efficient alternative to full projections in DNNs.
• This approach enables the development of highly accurate yet compact deep learning models.
• The findings facilitate the deployment of advanced DNNs on mobile and memory-limited systems.