Enhanced Computational Complexity in Continuous-Depth Models: Neural Ordinary Differential Equations With Trainable...

Area of Science:

• Machine Learning
• Numerical Analysis
• Dynamical Systems

Background:

• Neural Ordinary Differential Equations (NODEs) are continuous-time neural networks offering system-theoretic insights.
• NODEs are valuable for time series, forecasting, and invertible network applications.
• Current NODE training and inference are computationally intensive due to adaptive step-size solvers and high Number of Function Evaluations (NFE).

Purpose of the Study:

• To develop a novel approach for enhancing the computational efficiency of NODEs.
• To address the performance limitations of traditional adaptive step-size solvers in NODE models.
• To improve the practical applicability of NODEs through faster and more efficient computation.

Main Methods:

• Proposed making the parameters of the numerical integration scheme trainable within NODEs.
• Developed trainable fixed-step-size solvers that dynamically adapt to NODE dynamics.
• Compared the proposed trainable solvers against state-of-the-art methods like Dormand-Prince 5(4) (DOPRI).

Main Results:

• Achieved state-of-the-art performance across various benchmarks, including classification, density estimation, and dynamical system modeling.
• Demonstrated enhanced computational efficiency by operating with a fixed NFE.
• Showcased that trainable solvers adapt to NODE dynamics, improving overall performance.

Conclusions:

• Trainable numerical integration schemes offer a promising direction for efficient NODE modeling.
• The proposed method significantly improves computational efficiency while maintaining high accuracy.
• This work paves the way for more practical and widespread applications of NODEs.