An Efficient Algorithm for Deep Stochastic Contextual Bandits.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Optimization

Background:

• Stochastic contextual bandit (SCB) problems involve agents selecting actions based on context to maximize cumulative rewards.
• Deep neural networks (DNNs) are increasingly used to predict rewards in SCB, but convergence analysis is often overlooked.
• Existing methods lack theoretical guarantees for convergence in deep neural network-based SCB.

Purpose of the Study:

• To address the lack of convergence analysis in deep neural network-based SCB.
• To formulate SCB with DNN reward functions as a non-convex stochastic optimization problem.
• To design and analyze a novel algorithm for optimizing SCB problems with DNNs.

Main Methods:

• Formulated SCB with DNN reward functions as a non-convex stochastic optimization problem.
• Designed a stage-wise stochastic gradient descent algorithm for optimization.
• Provided theoretical convergence analysis for the proposed algorithm.

Main Results:

• The proposed algorithm converges to a greedy action policy respecting a local optimal reward function with high probability.
• Demonstrated the effectiveness and efficiency of the algorithm through extensive experiments.
• Validated the approach on multiple real-world datasets.

Conclusions:

• The developed stage-wise stochastic gradient descent algorithm provides theoretical convergence guarantees for SCB with DNNs.
• The algorithm is effective and efficient for optimizing policies in real-world SCB applications.
• This work bridges the gap between deep learning applications and theoretical convergence analysis in SCB.