Entropy Regularization in Deep Reinforcement Learning: A Structured Review Across Classical Control, Generative...

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Neuroscience

Background:

• Entropy regularization is a key mechanism in reinforcement learning (RL).
• Its interpretation and application vary significantly across different RL settings.
• Understanding these variations is crucial for advancing RL algorithms.

Purpose of the Study:

• To provide a unified taxonomy of entropy's role in various RL paradigms.
• To review mathematical foundations and recent advancements in entropy regularization.
• To clarify the nuanced benefits and drawbacks of entropy in different contexts.

Main Methods:

• Summarizing mathematical foundations of maximum-entropy RL, soft Bellman equations, and policy-gradient dynamics.
• Reviewing entropy applications in imitation learning, offline RL, intrinsic motivation, and generative policies.
• Highlighting recent work on entropy collapse in LLMs and novel entropy control mechanisms.

Main Results:

• Entropy's function shifts from exploration in online RL to ambiguity resolution in imitation learning.
• In offline RL, entropy requires balancing with data support; in LLMs, it relates to reasoning diversity and calibration.
• Recent methods address entropy collapse and introduce tractable entropy for flow-matching policies.

Conclusions:

• Entropy is not universally beneficial; its utility is context-dependent.
• Different applications require distinct entropy formulations and control strategies.
• Effective use of entropy necessitates careful consideration of exploration, data support, calibration, and reasoning diversity.