On Expressivity and Trainability of Quadratic Networks

Area of Science:

• Artificial intelligence
• Deep learning
• Computational neuroscience

Background:

• Biological neurons inspire quadratic artificial neurons, which replace inner-product operations with quadratic functions.
• Existing quadratic neural networks show promise but lack theoretical grounding and face training instability.

Purpose of the Study:

• To theoretically demonstrate the superior expressivity of quadratic neural networks.
• To develop a stable training strategy for quadratic neural networks.

Main Methods:

• Applied spline theory and algebraic geometry to prove enhanced model expressivity.
• Introduced referenced linear initialization (ReLinear) for stabilizing quadratic network training.

Main Results:

• Two theorems confirm quadratic networks possess greater model expressivity than conventional networks.
• The ReLinear strategy effectively stabilizes training, mitigating collapse risks.
• Experiments validate the enhanced performance of quadratic deep learning models.

Conclusions:

• Quadratic neural networks offer significant advantages in model expressivity.
• The ReLinear training strategy unlocks the full potential of quadratic deep learning.