Q-learning with temporal memory to navigate turbulence

Area of Science:

• Computational neuroscience
• Robotics
• Animal behavior

Background:

• Olfactory search is crucial for many organisms but challenging in turbulent environments.
• Previous models often rely on spatial cues or extensive prior knowledge.
• Understanding how agents navigate using only odor requires robust algorithms.

Purpose of the Study:

• To investigate if agents can learn robust olfactory navigation in turbulent environments using sequential decision-making.
• To develop and test a reinforcement learning algorithm for odor-guided navigation.
• To identify key olfactory features and memory requirements for successful navigation.

Main Methods:

• Developed a reinforcement learning algorithm utilizing interpretable olfactory states.
• Trained the agent using realistic turbulent odor cues and introduced temporal memory.
• Analyzed the impact of odor plume sparsity and agent-defined recovery strategies.

Main Results:

• Two salient olfactory features, discretized into few states, are sufficient for learning navigation.
• An optimal memory strategy was identified, ignoring odor blanks and employing a recovery strategy outside the plume.
• The learned recovery strategy, primarily crosswind casting, mirrors insect behavior and shows robustness to environmental changes.

Conclusions:

• Reinforcement learning with temporal memory enables robust olfactory navigation in turbulent environments.
• Learned strategies, including crosswind casting, are effective and adaptable.
• This approach provides insights into biological olfactory search and informs the design of autonomous search agents.