Recurrent Neural-Linear Posterior Sampling for Nonstationary Contextual Bandits

Area of Science:

• Machine Learning
• Artificial Intelligence
• Reinforcement Learning

Background:

• Nonstationary contextual bandit problems require agents to balance exploration and exploitation of changing patterns.
• Handcrafting historical context can transform nonstationary problems into stationary ones but may introduce spurious relationships or omit crucial information.

Purpose of the Study:

• To propose an approach that learns relevant context directly from raw interaction history, overcoming limitations of handcrafted contexts.
• To enhance decision-making in nonstationary environments by leveraging recurrent neural networks.

Main Methods:

• Utilizing recurrent neural networks (RNNs) to extract features from the raw history of agent-environment interactions.
• Combining RNN-extracted features with a contextual linear bandit algorithm employing posterior sampling.
• Evaluating the approach on diverse contextual and noncontextual nonstationary problems.

Main Results:

• The recurrent approach consistently outperformed feedforward counterparts that rely on handcrafted historical contexts.
• The proposed method demonstrated broader applicability compared to conventional nonstationary bandit algorithms.
• A novel regret bound for linear posterior sampling with measurement error was proven.

Conclusions:

• Learning context directly from raw interaction history using recurrent neural networks is a more effective and widely applicable strategy for nonstationary contextual bandit problems.
• The developed approach offers a promising direction for future theoretical work in bandit algorithms, particularly with the established regret bound.