A simple remedy for failure modes in physics informed neural networks

Area of Science:

• Computational Mathematics
• Machine Learning for Science
• Numerical Analysis

Background:

• Physics-informed neural networks (PINNs) are effective for solving partial differential equations (PDEs).
• PINNs face convergence challenges with complex PDEs, particularly those with large coefficients or high nonlinearity.
• A discrepancy in convergence rates between PDE and initial/boundary condition losses hinders PINN performance.

Purpose of the Study:

• To investigate the training dynamics of PINNs using neural tangent kernels (NTKs).
• To identify methods for improving PINN convergence and accuracy on complex PDEs.
• To analyze the impact of optimizers on the loss convergence discrepancy in PINNs.

Main Methods:

• Theoretical analysis of PINN training dynamics utilizing neural tangent kernels (NTKs).
• Investigating the effect of gradient descent with momentum (GDM) on loss convergence rates.
• Examining the role of the Adam optimizer in accelerating convergence and mitigating discrepancies.

Main Results:

• Gradient descent with momentum (GDM) significantly reduces the convergence rate gap between PDE and initial/boundary condition losses.
• The Adam optimizer also accelerates convergence and lessens the impact of the discrepancy.
• NTK analysis provides theoretical insights into why these optimizers enhance PINN performance.

Conclusions:

• PINNs trained with GDM or Adam exhibit improved convergence and accuracy for complex PDEs.
• The findings offer a pathway to enhance the robustness and applicability of PINNs in scientific computing.
• Neural tangent kernels are a valuable tool for understanding and improving deep learning models for PDEs.