Prospective Optimization.

Area of Science:

• Neuroscience
• Cognitive Science
• Artificial Intelligence

Background:

• Human performance in perceptual and motor tasks can approach ideal observer and actor limits.
• Optimal abilities depend on the cerebral cortex's capacity for information storage and rapid decision-making.
• Achieving these limits requires extensive learning through the interaction of the cerebral cortex and basal ganglia.

Purpose of the Study:

• To understand the algorithms the brain uses for reinforcement learning and prospective optimization.
• To explore the roles of the cerebral cortex and basal ganglia in goal-directed behavior.
• To identify general principles for developing autonomous devices.

Main Methods:

• Investigated algorithms for reinforcement learning, an online approximation of dynamic programming.
• Examined prospective optimization, where humans simulate scenarios based on past experience.
• Analyzed brain structure activity (cortex and basal ganglia) during both online optimal behavior and offline prospective optimization.

Main Results:

• The cerebral cortex and basal ganglia are crucial for learning sequences of actions toward goals.
• Reinforcement learning algorithms approximate dynamic programming in the brain.
• The same brain structures active during optimal behavior are also active during prospective optimization.

Conclusions:

• General principles and algorithms for goal-directed behavior are emerging from studies of the cortex-basal ganglia system.
• Understanding these principles has significant implications for the development of autonomous devices in engineering.
• The brain's mechanisms for learning and planning offer a blueprint for advanced AI.