Decentralized control and local information for robust and adaptive decentralized Deep Reinforcement Learning

Area of Science:

• Robotics
• Artificial Intelligence
• Computational Neuroscience

Background:

• Biological motor control utilizes decentralization for rapid, localized responses.
• Current Deep Reinforcement Learning (DRL) for motor control predominantly uses centralized architectures.
• Centralized DRL controllers must process extensive sensory information, potentially limiting efficiency.

Purpose of the Study:

• To investigate the benefits of decentralized control architectures in DRL for embodied sensori-motor control.
• To compare centralized versus decentralized DRL approaches for adaptive locomotion in a four-legged agent.
• To evaluate learning speed, robustness, and generalization capabilities across varying degrees of decentralization.

Main Methods:

• Developed and analyzed eight distinct control architectures for a four-legged agent.
• Systematically varied the degree of decentralization from fully centralized to fully decentralized.
• Assessed performance based on learning speed, robustness to hyperparameter settings, and generalization to untrained terrains.

Main Results:

• Distributed architectures significantly enhanced learning speed compared to centralized ones.
• Decentralized control demonstrated increased robustness, requiring less hyperparameter tuning and avoiding local minima.
• An intermediate decentralized architecture, integrating local information from neighboring legs, showed superior generalization to uneven terrains.

Conclusions:

• Distributing control into decentralized units leveraging local information offers significant advantages for DRL-based motor control.
• Decentralization improves learning efficiency, robustness, and generalization capabilities in adaptive locomotion tasks.
• This approach presents a promising direction for developing more resilient and adaptable DRL systems.