Statistical Inference for Online Decision-Making: In a Contextual Bandit Setting.

Area of Science:

• Machine Learning
• Online Decision-Making

Background:

• Online decision-making involves sequential choices based on evolving information.
• Common approaches learn reward models to maximize long-term gains.
• Assessing model reasonableness and asymptotic performance is crucial.

Purpose of the Study:

• To analyze the asymptotic properties of estimators in a linear contextual bandit framework.
• To address model misspecification in online decision-making.
• To provide theoretical guarantees for online learning algorithms.

Main Methods:

• Utilized the martingale central limit theorem for theoretical analysis.
• Employed an epsilon-greedy policy for exploration-exploitation.
• Developed online ordinary least squares and weighted least squares estimators.
• Applied inverse propensity score weighting for misspecified models.

Main Results:

• Established asymptotic normality for the online ordinary least squares estimator of model parameters.
• Demonstrated asymptotic normality for the online weighted least squares estimator under model misspecification.
• Proved asymptotic normality for the in-sample inverse propensity weighted value estimator.

Conclusions:

• The proposed estimators provide reliable performance evaluation in online decision-making.
• Theoretical guarantees are established for linear contextual bandit models.
• The findings are validated through simulations and a real-world news recommendation dataset.