Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Randomized Experiments01:13

Randomized Experiments

6.9K
The randomization process involves assigning study participants randomly to experimental or control groups based on their probability of being equally assigned. Randomization is meant to eliminate selection bias and balance known and unknown confounding factors so that the control group is similar to the treatment group as much as possible. A computer program and a random number generator can be used to assign participants to groups in a way that minimizes bias.
Simple randomization
Simple...
6.9K
Observational Learning01:12

Observational Learning

163
Albert Bandura's observational learning, also known as imitation or modeling, occurs when a person observes and imitates another's behavior. It is a quicker process than operant conditioning. A well-known example is the Bobo doll study, where children who saw an adult acting aggressively towards the doll were more likely to act aggressively when left alone, compared to those who observed a nonaggressive adult. Many psychologists view observational learning as a form of latent learning...
163
Trial and Error and Algorithm01:12

Trial and Error and Algorithm

108
A problem-solving strategy is a plan of action used to find a solution. Different strategies have distinct action plans. Trial and error involves trying different solutions until one works. For instance, to fix a broken printer, you might check ink levels, ensure the paper tray isn't jammed, and verify the printer's connection to your laptop. This method can be time-consuming but is commonly used. Thomas Edison, for example, used trial and error to find a suitable filament for the light...
108
Prediction Intervals01:03

Prediction Intervals

2.3K
The interval estimate of any variable is known as the prediction interval. It helps decide if a point estimate is dependable.
However, the point estimate is most likely not the exact value of the population parameter, but close to it. After calculating point estimates, we construct interval estimates, called confidence intervals or prediction intervals. This prediction interval comprises a range of values unlike the point estimate and is a better predictor of the observed sample value, y. 
2.3K
Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving01:29

Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving

48
Mechanistic models play a crucial role in algorithms for numerical problem-solving, particularly in nonlinear mixed effects modeling (NMEM). These models aim to minimize specific objective functions by evaluating various parameter estimates, leading to the development of systematic algorithms. In some cases, linearization techniques approximate the model using linear equations.
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
48
Law of Effect01:06

Law of Effect

1.4K
B.F. Skinner, a prominent figure in behavioral psychology, introduced operant conditioning by emphasizing the role of consequences in shaping behavior. This theory builds upon the law of effect proposed by Edward Thorndike, which posits that behaviors followed by satisfying outcomes are likely to be repeated. In contrast, those followed by unsatisfying outcomes are less likely to recur.
Edward Thorndike's foundational work involved studying learning in animals, particularly using puzzle...
1.4K

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Loss of plasticity in deep continual learning.

Nature·2024
Same author

Communicative capital: a key resource for human-machine shared agency and collaborative capacity.

Neural computing & applications·2023
Same author

From eye-blinks to state construction: Diagnostic benchmarks for online representation learning.

Adaptive behavior·2023
Same author

Reactive Reinforcement Learning in Asynchronous Environments.

Frontiers in robotics and AI·2021
Same author

Pavlovian control of intraspinal microstimulation to produce over-ground walking.

Journal of neural engineering·2020
Same author

Using Functional or Structural Magnetic Resonance Images and Personal Characteristic Data to Identify ADHD and Autism.

PloS one·2016

Related Experiment Video

Updated: Jun 24, 2025

Measuring Statistical Learning Across Modalities and Domains in School-Aged Children Via an Online Platform and Neuroimaging Techniques
08:05

Measuring Statistical Learning Across Modalities and Domains in School-Aged Children Via an Online Platform and Neuroimaging Techniques

Published on: June 30, 2020

7.5K

Off-Policy Prediction Learning: An Empirical Study of Online Algorithms.

Sina Ghiassian, Banafsheh Rafiee, Richard S Sutton

    IEEE Transactions on Neural Networks and Learning Systems
    |June 10, 2024
    PubMed
    Summary

    This study compares 11 off-policy prediction algorithms in reinforcement learning, finding Emphatic TD(lambda) has low error but can be slow. New Ratchet algorithms improve learning speed by adapting step-size dynamically.

    More Related Videos

    Using Eye Movements Recorded in the Visual World Paradigm to Explore the Online Processing of Spoken Language
    09:27

    Using Eye Movements Recorded in the Visual World Paradigm to Explore the Online Processing of Spoken Language

    Published on: October 13, 2018

    10.0K
    Methods to Test Visual Attention Online
    09:44

    Methods to Test Visual Attention Online

    Published on: February 19, 2015

    11.8K

    Related Experiment Videos

    Last Updated: Jun 24, 2025

    Measuring Statistical Learning Across Modalities and Domains in School-Aged Children Via an Online Platform and Neuroimaging Techniques
    08:05

    Measuring Statistical Learning Across Modalities and Domains in School-Aged Children Via an Online Platform and Neuroimaging Techniques

    Published on: June 30, 2020

    7.5K
    Using Eye Movements Recorded in the Visual World Paradigm to Explore the Online Processing of Spoken Language
    09:27

    Using Eye Movements Recorded in the Visual World Paradigm to Explore the Online Processing of Spoken Language

    Published on: October 13, 2018

    10.0K
    Methods to Test Visual Attention Online
    09:44

    Methods to Test Visual Attention Online

    Published on: February 19, 2015

    11.8K

    Area of Science:

    • Reinforcement Learning
    • Machine Learning
    • Artificial Intelligence

    Background:

    • Off-policy prediction, crucial for reinforcement learning, involves learning a value function from data generated by a different policy.
    • Existing off-policy prediction algorithms face challenges with learning speed, parameter sensitivity, and asymptotic error, especially in high-variance scenarios.
    • The impact of importance sampling ratios on algorithm performance and the need for effective step-size adaptation remain critical research areas.

    Purpose of the Study:

    • To empirically evaluate and compare the performance of 11 distinct off-policy prediction learning algorithms.
    • To analyze algorithm sensitivity to parameters, learning speed, and asymptotic error across various tasks, including those with high variance.
    • To propose novel step-size adaptation methods, the Ratchet algorithms, to enhance the learning speed of the best-performing algorithm.

    Main Methods:

    • Empirical comparison of 11 off-policy prediction algorithms using linear function approximation on three distinct tasks: Collision, Rooms, and High Variance Rooms.
    • Analysis of algorithm performance based on sensitivity to parameters, learning speed, and asymptotic error, with a focus on variance induced by importance sampling ratios.
    • Development and proposal of two new step-size adaptation methods: Step-size Ratchet and Soft Step-size Ratchet.

    Main Results:

    • Algorithm performance is significantly influenced by the variance introduced by importance sampling ratios.
    • Tree Backup, Vtrace, and ABTD show resilience to high variance but impose limiting constraints on the bootstrapping parameter.
    • Emphatic TD(lambda) demonstrates lower asymptotic error but potentially slower learning, while the proposed Ratchet algorithms effectively improve learning speed.

    Conclusions:

    • The choice of off-policy prediction algorithm critically impacts performance, especially in high-variance environments.
    • The proposed Ratchet algorithms offer an effective solution for accelerating learning by dynamically adjusting step-sizes.
    • Further research into adaptive step-size strategies is crucial for advancing the efficiency and applicability of off-policy prediction methods.