Search research articles

ABOUT JoVE

Overview Leadership Blog JoVE Help Center

AUTHORS

Publishing Process Editorial Board Scope & Policies Peer Review FAQ Submit

LIBRARIANS

Testimonials Subscriptions Access Resources Library Advisory Board FAQ

RESEARCH

JoVE Journal Methods Collections JoVE Encyclopedia of Experiments Archive

EDUCATION

JoVE Core JoVE Business JoVE Science Education JoVE Lab Manual Faculty Resource Center Faculty Site

Terms & Conditions of Use

Related Concept Videos

Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving

Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving

Mechanistic models play a crucial role in algorithms for numerical problem-solving, particularly in nonlinear mixed effects modeling (NMEM). These models aim to minimize specific objective functions by evaluating various parameter estimates, leading to the development of systematic algorithms. In some cases, linearization techniques approximate the model using linear equations.
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...

State Space Representation

State Space Representation

The frequency-domain technique, commonly used in analyzing and designing feedback control systems, is effective for linear, time-invariant systems. However, it falls short when dealing with nonlinear, time-varying, and multiple-input multiple-output systems. The time-domain or state-space approach addresses these limitations by utilizing state variables to construct simultaneous, first-order differential equations, known as state equations, for an nth-order system.
Consider an RLC circuit, a...

Mechanistic Models: Overview of Compartment Models

Mechanistic Models: Overview of Compartment Models

Mechanistic models, a category encompassing both physiological and compartmental modeling, differ from empirical models' approaches to incorporating known factors about the systems being modeled. Empirical models describe data with minimal assumptions, while mechanistic models aim to provide a robust description of available data by specifying assumptions and integrating known factors about the system. Compartmental analysis is a key example of a mechanistic model in pharmacokinetics and...

Comparison between RL and RC circuits

Comparison between RL and RC circuits

An RC circuit consists of resistance and capacitance, while in an RL circuit, capacitance is replaced by an inductor. RL and RC circuits are first-order differential circuits that store energy. An RC circuit stores energy in the electric field, while an RL circuit stores energy in the magnetic field. When connected to a battery, an RC circuit charges the capacitor, causing the current to decrease from maximum to zero upon being fully charged. This increases the voltage across the capacitor from...

Multicompartment Models: Overview

Multicompartment Models: Overview

Multicompartment models are mathematical constructs that depict how drugs are distributed and eliminated within the body. They segment the body into several compartments, symbolizing various physiological or anatomical areas connected through drug transfer processes such as absorption, metabolism, distribution, and elimination.
These models offer a more comprehensive representation of drug behavior in the body than one-compartment models. They accommodate the complexity of drug distribution,...

Multi-input and Multi-variable systems

Multi-input and Multi-variable systems

Cruise control systems in cars are designed as multi-input systems to maintain a driver's desired speed while compensating for external disturbances such as changes in terrain. The block diagram for a cruise control system typically includes two main inputs: the desired speed set by the driver and any external disturbances, such as the incline of the road. By adjusting the engine throttle, the system maintains the vehicle's speed as close to the desired value as possible.
In the absence of...

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by

Same author

The emerging picture of a complex core-mantle boundary.

Nature communications·2024

Same author

Managing extreme AI risks amid rapid progress.

Science (New York, N.Y.)·2024

Same author

Regulating advanced artificial agents.

Science (New York, N.Y.)·2024

Same author

Interesting case of regional left ventricular hypertrophy.

BMJ case reports·2024

Same author

The 2023 International Society for Heart and Lung Transplantation Guidelines for Mechanical Circulatory Support: A 10- Year Update.

The Journal of heart and lung transplantation : the official publication of the International Society for Heart Transplantation·2023

Same author

AI weapons: Russia's war in Ukraine shows why the world must enact a ban.

Nature·2023

Search research articles

Related Experiment Video

Updated: May 15, 2026

WheelCon: A Wheel Control-Based Gaming Platform for Studying Human Sensorimotor Control

WheelCon: A Wheel Control-Based Gaming Platform for Studying Human Sensorimotor Control

Published on: August 15, 2020

On the representation complexity of model-based and model-free reinforcement learning.

Hanlin Zhu¹, Baihe Huang¹, Stuart Russell¹

¹EECS, University of California, Berkeley, Berkeley, CA, USA.

Philosophical Transactions. Series A, Mathematical, Physical, and Engineering Sciences

|May 14, 2026

Summary

Model-based reinforcement learning (RL) benefits from simpler environment models, unlike model-free RL. This representation complexity explains why model-based methods often require less data for learning complex tasks.

Keywords:

circuit complexity model-based reinforcement learning representation complexity sample complexity world model

Related Experiment Videos

Last Updated: May 15, 2026

WheelCon: A Wheel Control-Based Gaming Platform for Studying Human Sensorimotor Control

WheelCon: A Wheel Control-Based Gaming Platform for Studying Human Sensorimotor Control

Published on: August 15, 2020

Area of Science:

Artificial Intelligence
Machine Learning
Computational Complexity Theory

Background:

Reinforcement learning (RL) algorithms are broadly categorized into model-based and model-free approaches.
Understanding the representation complexity of these algorithms is crucial for improving sample efficiency and performance.
Prior research has not extensively explored the link between circuit complexity and the differing sample complexities of RL paradigms.

Purpose of the Study:

To investigate the representation complexity of model-based and model-free reinforcement learning using circuit complexity theory.
To provide theoretical insights into why model-based RL algorithms often exhibit superior sample complexity compared to model-free methods.
To establish a rigorous framework for analyzing the representational demands of RL components.

Main Methods:

Analyzing the circuit complexity of transition and reward functions in Markov decision processes.
Comparing the circuit complexity of optimal Q-functions with that of environment models.
Empirically evaluating approximation errors of transition kernels, reward functions, and Q-functions in MuJoCo environments.

Main Results:

Demonstrated that environment transition and reward functions can be represented by polynomial-sized, constant-depth circuits.
Showed that optimal Q-functions require exponential circuit complexity in constant-depth models.
Empirical results confirmed consistently lower approximation errors for transition kernels and reward functions compared to optimal Q-functions.

Conclusions:

The inherent simplicity in representing environment dynamics (world models) contributes to the sample efficiency of model-based RL.
Model-free RL's reliance on complex Q-functions, which are harder to represent, leads to higher sample complexity.
This research highlights the significance of world models in AI and offers a novel perspective on RL algorithm design and analysis.