Adaptive parameter manifold learning for low-dimensional neural network optimization

Deep neural networks have achieved remarkable success across diverse domains, yet their massive over-parameterization introduces significant optimization challenges, particularly susceptibility to overfitting and noise memorization. While traditional stochastic gradient descent treats all parameters equally in the native parameter space, recent evidence suggests that neural network optimization occurs within much lower-dimensional subspaces. Existing parameter subspace methods either rely on random projections that ignore parameter space structure or employ linear dimensionality reduction that cannot capture complex geometric relationships. Through empirical analysis of training trajectories, we demonstrate that neural network parameters evolve along continuous, low-dimensional non-linear manifolds whose intrinsic dimensionality is substantially lower than the original parameter space. We propose Adaptive Parameter Manifold Learning (APML), a novel optimization framework that leverages Local Tangent Space Alignment to discover the underlying manifold structure from training trajectories and optimize neural networks on the parameter manifolds. Our method dynamically adapts the manifold throughout training to accommodate the changing geometry of the loss landscape. Extensive experiments on noisy label learning tasks across vision and language domains, as well as federated learning scenarios, demonstrate that APML achieves superior robustness and communication efficiency compared to existing subspace optimization methods.