Gauss-Newton Temporal Difference Learning With Nonlinear Function Approximation.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Q-learning is a fundamental reinforcement learning algorithm.
• Nonlinear function approximations are crucial for handling complex state spaces in Q-learning.
• Existing temporal difference (TD) methods face challenges with sample complexity and convergence rates.

Purpose of the Study:

• To propose a novel Gauss-Newton temporal difference (GNTD) learning method.
• To address the limitations of existing methods in terms of sample complexity and convergence.
• To enable efficient and stable Q-learning with nonlinear function approximations.

Main Methods:

• The proposed GNTD method utilizes Gauss-Newton (GN) steps to optimize a variant of mean-squared Bellman error (MSBE).
• Target networks are employed to mitigate issues related to double sampling.
• Inexact GN steps are analyzed for efficient computation using matrix iterations.
• Nonasymptotic finite-sample convergence guarantees are derived under mild conditions.

Main Results:

• GNTD achieves an improved sample complexity of $\tilde {\mathcal {O}}(\varepsilon ^{-1})$ for neural networks with ReLU activation, outperforming existing TD methods.
• For general smooth function approximations, GNTD establishes a sample complexity of $\tilde {\mathcal {O}}(\varepsilon ^{-1.5})$ .
• Extensive experiments on reinforcement learning benchmarks demonstrate that GNTD yields higher rewards and faster convergence compared to TD-type methods.

Conclusions:

• The GNTD learning method provides a theoretically sound and empirically effective approach for Q-learning with nonlinear function approximations.
• GNTD significantly enhances sample efficiency and convergence speed, particularly in deep reinforcement learning settings.
• The proposed method offers a promising advancement for tackling complex reinforcement learning problems.