Deep dive into model-free reinforcement learning for underwater locomotion: theory and practice

Area of Science:

• Robotics
• Computational Neuroscience
• Artificial Intelligence

Background:

• Aquatic animals and underwater robots require sophisticated sensorimotor control for tasks like navigation and predation in complex environments.
• Deep reinforcement learning (RL) offers a powerful framework for synthesizing control policies for embodied agents in simulated physical worlds.

Purpose of the Study:

• To provide a self-contained introduction to model-free reinforcement learning for embodied agents in underwater environments.
• To highlight the role of physical modeling choices in formulating RL problems.
• To offer guidelines for applying RL to understand animal behavior and design bioinspired robots.

Main Methods:

• Focuses on actor-critic methods within model-free reinforcement learning.
• Presents the mathematical formulation of RL, emphasizing physical modeling.
• Discusses practical implementation aspects of actor-critic algorithms.

Main Results:

• Provides examples of RL-controlled swimmers.
• Offers guidelines for selecting observations, actions, and rewards aligned with biological behavior.
• Demonstrates RL's utility in exploring hypotheses about feedback control in biological and robotic systems.

Conclusions:

• Reinforcement learning provides a versatile framework for studying sensorimotor control in both natural and artificial aquatic systems.
• This tutorial equips researchers with the foundational knowledge and practical considerations for applying RL to embodied underwater agents.