Exact learning dynamics of deep linear networks with prior knowledge.

Area of Science:

• Machine Learning
• Deep Learning Theory
• Computational Neuroscience

Background:

• Deep neural network (DNN) learning effectiveness is heavily influenced by initial network weights.
• Theoretical understanding of how prior knowledge in initial weights shapes learning dynamics remains limited.

Purpose of the Study:

• To derive exact solutions for learning dynamics in deep linear networks with rich prior knowledge.
• To generalize existing theoretical frameworks for analyzing learning in DNNs.

Main Methods:

• Generalization of Fukumizu's matrix Riccati solution.
• Derivation of explicit expressions for evolving network function, representational similarity, and neural tangent kernel.
• Analysis of a broad class of initializations and tasks.

Main Results:

• Identified task-independent initializations that accelerate learning dynamics from slow to fast exponential trajectories.
• Demonstrated convergence to a global optimum with preserved representational similarity, independent of initial internal representations.
• Characterized the dynamic alignment of network weights with task structure.

Conclusions:

• Developed a mathematical toolkit for understanding the impact of prior knowledge on deep learning dynamics.
• Provided rigorous justification for previous learning models and highlighted implications for continual and reversal learning.
• Showcased how specific initializations can decouple learning trajectories from initial representational structures.