Orthogonal Deep Neural Networks.

Area of Science:

• Machine Learning
• Deep Learning Theory
• Artificial Intelligence

Background:

• Deep Neural Networks (DNNs) generalization is a key research area.
• Spectrally regularized deep learning methods are gaining interest.
• Understanding network weight properties is crucial for improved generalization.

Purpose of the Study:

• Introduce Orthogonal Deep Neural Networks (OrthDNNs) algorithms.
• Develop theoretically motivated methods for better DNN generalization.
• Establish a new generalization error bound sensitive to weight matrix spectral properties.

Main Methods:

• Prove DNNs exhibit local isometry on practical data distributions.
• Derive a new generalization error bound using sample space covering and local isometry.
• Propose strict and approximate OrthDNN algorithms, including Singular Value Bounding (SVB).
• Introduce Bounded Batch Normalization (BBN) for compatibility with batch normalization.

Main Results:

• Demonstrate that weight matrices with equal singular values optimize the generalization bound.
• Show that orthogonal matrices or matrices with orthonormal rows/columns are optimal choices.
• SVB algorithm achieves performance comparable to strict OrthDNNs at lower computational cost.
• Extensive experiments on image classification confirm the efficacy of OrthDNNs.

Conclusions:

• OrthDNNs offer a theoretically grounded approach to enhance deep learning generalization.
• The proposed SVB and BBN methods provide practical and efficient ways to implement OrthDNNs.
• Orthogonal Deep Neural Networks represent a promising direction for developing more robust and generalizable AI models.