Cooperative Deep Q-Learning Framework for Environments Providing Image Feedback

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Reinforcement Learning

Background:

• Deep reinforcement learning (DRL) faces challenges with sample inefficiency and slow learning processes.
• Approximating the action-value function, especially with image inputs, remains a complex task in DRL.

Purpose of the Study:

• To address sample inefficiency and slow learning in DRL using a dual-neural network (NN) approach.
• To develop a novel error-driven learning (EDL) method for robust action-value function approximation.

Main Methods:

• Employed a dual-NN architecture with independent initialization for approximating the action-value function.
• Introduced a temporal difference (TD) error-driven learning (EDL) approach with linear transformations for direct NN layer updates.
• Provided theoretical analysis demonstrating the EDL regime's cost minimization and error reduction over learning progression.

Main Results:

• The EDL approach theoretically approximates the empirical cost, with decreasing approximation error as learning progresses.
• Simulation analysis confirmed that the proposed dual-NN and EDL methods lead to faster learning and convergence.
• The methods demonstrated a reduction in required buffer size, significantly enhancing sample efficiency.

Conclusions:

• The dual-NN driven EDL approach effectively tackles key challenges in DRL, namely sample inefficiency and slow learning.
• The proposed EDL method offers a robust and efficient way to update deep neural network parameters in DRL.
• This research contributes to more efficient and faster training of DRL agents, particularly in environments with visual inputs.