Related Experiment Video
Updated: Jun 4, 2025
10:19
Studying the Neural Basis of Adaptive Locomotor Behavior in Insects
Published on: April 13, 2011
12.8K
Neurocontrol for fixed-length trajectories in environments with soft barriers.
Michael Fairbank1, Danil Prokhorov2, David Barragan-Alcantar1
1School of Computer Science and Electronic Engineering, University of Essex, Colchester, UK.
Summary
Training simulated agents with analytic policy gradients can lead to learning issues. Using soft barriers and gradient clipping or truncation helps avoid local minima and exploding gradients for better neurocontrol.
Area of Science:
- Neuroscience
- Machine Learning
- Reinforcement Learning
Background:
- Analytic policy gradient with back-propagation through time is used for training simulated agents.
- Terminal barriers in environments can cause learning to get stuck in local minima or oscillating limit cycles.
Purpose of the Study:
- To address learning challenges in neurocontrol problems with terminal barriers.
- To propose and evaluate methods for improving agent training stability and performance.
Main Methods:
- Implementing fixed-length trajectories and soft barriers with penalty costs.
- Investigating the impact of soft barriers on learning gradients.
- Applying gradient clipping and smooth truncation of gradients.
Main Results:
- Soft barriers can cause exploding learning gradients, often at inappropriate trajectory points.
- Exploding gradients combined with adaptive optimizers can halt learning.
- Gradient clipping and smooth truncation effectively mitigate exploding gradients and improve learning focus.
Conclusions:
- Soft barriers offer a way to avoid local minima but introduce gradient issues.
- Careful gradient management techniques are crucial for stable and effective neurocontrol training.
- The proposed methods enable agents to learn more effectively by focusing on relevant trajectory parts.
