TBC-HRL: A Bio-Inspired Framework for Stable and Interpretable Hierarchical Reinforcement Learning

Area of Science:

• Artificial Intelligence
• Computational Neuroscience
• Robotics

Background:

• Hierarchical Reinforcement Learning (HRL) effectively decomposes complex tasks but faces challenges like inter-level instability and poor interpretability.
• Real-world applications of HRL are hindered by issues such as inefficient subgoal scheduling and delayed responses.

Purpose of the Study:

• To introduce Timed and Bionic Circuit Hierarchical Reinforcement Learning (TBC-HRL), a novel framework addressing HRL limitations.
• To enhance coordination, consistency, and temporal dependency modeling in reinforcement learning agents.

Main Methods:

• Implemented a timed subgoal scheduling strategy with fixed execution durations (τ) to mimic rhythmic biological action patterns.
• Introduced a Neuro-Dynamic Bionic Circuit Network (NDBCNet), inspired by C. elegans, for low-level control, featuring sparse connectivity and continuous-time dynamics.

Main Results:

• TBC-HRL demonstrated improved policy stability and action precision across six complex simulated tasks.
• The NDBCNet component offered enhanced interpretability and reduced computational overhead compared to traditional networks.
• Experiments confirmed TBC-HRL's superior adaptability in dynamic environments.

Conclusions:

• Biologically inspired mechanisms, like timed scheduling and bionic circuits, significantly advance HRL capabilities.
• TBC-HRL presents a practical and potentially transformative approach for intelligent control systems, especially on resource-constrained platforms.