A Unifying Framework for Reinforcement Learning and Planning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Operations Research

Background:

• Sequential decision making is often formalized as Markov Decision Process (MDP) optimization.
• Reinforcement learning (RL) and planning are two distinct yet successful approaches to MDP optimization.
• These fields have largely developed independently, despite addressing the same core problem.

Purpose of the Study:

• To present a unifying algorithmic framework for reinforcement learning and planning (FRAP).
• To identify underlying decision dimensions common to both MDP planning and learning algorithms.
• To provide a basis for deeper insight into the algorithmic design space for MDPs.

Main Methods:

• Developed a unifying algorithmic framework (FRAP) for RL and planning.
• Identified key decision dimensions relevant to MDP optimization algorithms.
• Compared existing planning, model-free RL, and model-based RL algorithms using the FRAP dimensions.

Main Results:

• The FRAP framework successfully disentangles common factors in planning and learning approaches.
• A comparative analysis of various algorithms along the identified dimensions is presented.
• The framework offers a structured way to understand and design MDP optimization algorithms.

Conclusions:

• The proposed FRAP framework provides a unified perspective on reinforcement learning and planning.
• This unification facilitates deeper understanding and potential innovation in designing algorithms for sequential decision making.
• The comparative analysis highlights the strengths and weaknesses of different approaches within the unified framework.