Learning in PINNs: Phase transition, diffusion equilibrium, and generalization.

The optimization of non-convex objectives in deep learning relies on understanding how first-order optimizers navigate complex loss landscapes to reach global or local minima. It was already known that the learning process involves distinct drift and diffusion phases as described by information bottleneck theory, which posits a trade-off between data fitting and representation compression. These phases characterize how networks manage the flow of information through successive layers while attempting to minimize empirical risk. Existing models often struggle to maintain stable convergence when sample-wise gradients exhibit high variance or significant misalignment across the training set. Researchers frequently observe that stochastic gradient descent behaves differently depending on the signal quality within the batch, leading to unpredictable generalization outcomes. Understanding the transition between these noisy and stable states remains a significant challenge for designing robust architectures in scientific computing. This absence of evidence motivated a deeper investigation into the specific conditions that define stable training regimes beyond the initial diffusion stage.

This research investigates the learning dynamics of fully-connected neural networks (FCNN) by analyzing the neural gradient Signal-to-noise Ratio (SNR) throughout the training trajectory. The study seeks to identify unique phase transitions that occur during the training of first-order optimizers within non-convex objective functions. Researchers aimed to define the characteristics of a stable state termed Diffusion Equilibrium (DE), which follows the traditional drift and diffusion stages. Another objective involves examining how homogeneous residuals across the sample space influence model generalization and optimization sensitivity. The team also explored the relationship between information compression and activation saturation during these specific phase transitions. They specifically focused on how gradient directional alignment drives the saturation of internal neurons and affects the overall model convergence. Finally, the work evaluates these phenomena within the context of Physics-Informed Neural Networks (PINNs) to address the inherent Partial Differential Equation (PDE) interdependencies between samples.

The investigators utilized fully-connected neural networks (FCNN) to monitor gradient behavior and signal quality throughout the entire optimization process. They calculated the neural gradient Signal-to-noise Ratio (SNR) to quantify the alignment of sample-wise updates across the high-dimensional parameter space. A novel sample-wise re-weighting scheme was implemented to target problematic training points characterized by large residuals and vanishing gradients. This re-weighting approach specifically addressed quadratic loss functions to improve residual homogeneity and ensure equal sensitivity to each training sample. Information compression was measured by analyzing activation patterns and saturation levels across different network layers during the transition to the equilibrium phase. The experimental validation focused on Physics-Informed Neural Networks (PINNs) where sample interdependence is strictly dictated by underlying physical laws and differential operators. Researchers analyzed the first-order transition points to determine when the system enters a highly-ordered state characterized by gradient agreement.

The identification of the Diffusion Equilibrium (DE) phase revealed a stable training period marked by highly-ordered neural gradients across the entire sample space. An abrupt first-order transition occurred where sample-wise gradients aligned and the Signal-to-noise Ratio (SNR) increased significantly, signaling stable optimizer convergence. Achieving homogeneous residuals during this specific phase directly correlated with enhanced generalization, as optimization steps became equally sensitive to every training sample. The proposed re-weighting scheme successfully reduced residuals for samples that previously exhibited vanishing gradients, thereby improving the overall training stability. Activation saturation at the phase transition induced significant information compression while maintaining negligible information loss in the deeper layers of the network. Models demonstrated faster convergence when both sample-wise gradients and residuals transitioned into an ordered state, facilitating more efficient learning. Experimental data from PINNs confirmed that gradient agreement is essential due to the inherent interdependence of samples in models constrained by physical equations.

These findings suggest that monitoring phase transitions can significantly refine deep learning optimization strategies for complex scientific and engineering problems. The discovery of the Diffusion Equilibrium (DE) phase provides a new framework for understanding stable optimizer behavior in non-convex landscapes. Enhancing residual homogeneity offers a practical pathway to improving the performance and reliability of Physics-Informed Neural Networks (PINNs). Future optimization techniques might leverage gradient alignment to ensure consistent sensitivity across all training samples, preventing the model from ignoring difficult data points. The observed saturation-induced compression highlights how architectural depth preserves essential information during convergence while discarding redundant noise. This research establishes a vital link between information theory and the practical training of networks solving complex physical problems involving differential equations. Identifying these specific transitions allows for more predictable and efficient machine learning performance across a wide range of technical applications.