Counterfactual-Augmented Importance Sampling for Semi-Offline Policy Evaluation

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Statistics

Background:

• Off-policy evaluation (OPE) using observational data is limited by distribution shifts.
• Online evaluation is often infeasible in high-stakes domains due to safety concerns.

Purpose of the Study:

• To propose a semi-offline evaluation framework for reinforcement learning (RL).
• To incorporate human annotations of counterfactual trajectories to improve OPE.
• To develop novel OPE estimators that mitigate bias and variance.

Main Methods:

• Developed a semi-offline evaluation framework combining offline data with human annotations.
• Designed a new family of OPE estimators using importance sampling (IS) and a novel weighting scheme.
• Analyzed theoretical properties and conducted proof-of-concept experiments.

Main Results:

• The proposed method incorporates counterfactual annotations without introducing bias.
• The approach demonstrated potential for reducing both bias and variance compared to standard IS estimators.
• Experiments showed superior performance over purely offline IS estimators, even with imperfect annotations.

Conclusions:

• The semi-offline framework enables safer and more reliable RL policy evaluation in critical applications.
• Human-centered annotation design is crucial for effective implementation.
• This work facilitates RL adoption in high-stakes domains by addressing evaluation challenges.