Inverse Rational Control with Partially Observable Continuous Nonlinear Dynamics.

Area of Science:

• Neuroscience
• Computational Neuroscience
• Reinforcement Learning

Background:

• The brain uses internal models to process ambiguous sensory data for action guidance.
• Reinforcement learning under partial observations is a key computational framework for decision-making.
• Animals often exhibit behavior that appears suboptimal, posing a challenge to understanding their decision-making processes.

Purpose of the Study:

• To develop a generalized Inverse Rational Control (IRC) method for continuous nonlinear systems.
• To identify the internal world models that best explain observed animal behavior, even if suboptimal.
• To accommodate sensory observations corrupted by private, unknown noise.

Main Methods:

• An optimal Bayesian agent was trained using deep reinforcement learning over a model space of dynamics and rewards.
• A likelihood function was derived to assess models based on observed action trajectories from suboptimal agents.
• Gradient ascent was employed to maximize the likelihood and identify the most probable internal model.

Main Results:

• The generalized IRC method successfully recovered the true internal models of rational agents.
• The approach accommodates continuous nonlinear dynamics, continuous actions, and noisy sensory inputs.
• The method provides a computational framework for inferring internal models from behavior.

Conclusions:

• Animal behavior can be understood as rational given a flawed internal model, not necessarily suboptimal.
• This work extends Inverse Rational Control to more complex, realistic scenarios.
• The developed method offers a foundation for interpreting neural and behavioral dynamics in decision-making tasks.