Do Neural Network Weights Account for Classes Centers?

Area of Science:

• Machine Learning
• Deep Learning
• Computer Vision

Background:

• Deep neural networks (DNNs) are increasingly used for feature learning, focusing on loss functions for class discrimination and feature vector cohesion.
• Current methods often assume class weight vectors represent geometrical centers in the feature space, a hypothesis this study investigates.
• This assumption can lead to instability issues during DNN training.

Purpose of the Study:

• To theoretically analyze the geometrical assumptions in DNN feature learning.
• To identify and address the stability issues arising from these assumptions.
• To propose and validate a novel symmetry for weight vectors to improve DNN training stability.

Main Methods:

• Theoretical analysis of feature space geometry in DNNs.
• Empirical validation of training stability with and without the proposed symmetry.
• Analytical study of a specific weight vector symmetry (unit, coplanar, zero sum).

Main Results:

• The empirical assumption about class weight vectors as geometrical centers is not always met.
• The proposed weight vector symmetry (unit, coplanar, zero sum) addresses DNN training stability issues.
• The novel symmetry results in a more stable learning curve compared to existing models.

Conclusions:

• The geometrical assumptions in current feature learning DNNs can compromise training stability.
• A specific weight vector symmetry is proposed and validated to ensure stable DNN training.
• This approach offers a more robust method for feature learning with deep neural networks.