Simple modification of the upper confidence bound algorithm by generalized weighted averages.

Area of Science:

• Reinforcement Learning
• Machine Learning
• Artificial Intelligence

Background:

• The multi-armed bandit (MAB) problem is a fundamental challenge in reinforcement learning, focusing on sequential decision-making under uncertainty.
• Existing algorithms like UCB1 provide a baseline for balancing exploration and exploitation, but advancements are needed for diverse applications.

Purpose of the Study:

• To introduce a novel generalized Upper Confidence Bound algorithm, GWA-UCB1, designed to improve upon the UCB1 algorithm for MAB problems.
• To provide a flexible and easily implementable algorithm that generalizes the exploration-exploitation trade-off in UCB1.
• To evaluate the performance of GWA-UCB1 across various stochastic and survival MAB problem settings.

Main Methods:

• The study proposes the GWA-UCB1 algorithm, which extends UCB1 using generalized weighted averages.
• Preliminary experiments involved investigating optimal parameters for GWA-UCB1 and a simpler G-UCB1 variant.
• Algorithm performance was validated on stochastic MAB problems with uniform/normal reward distributions and survival MAB problems.

Main Results:

• GWA-UCB1 demonstrated superior performance compared to G-UCB1, UCB1-Tuned, and Thompson sampling across most tested scenarios.
• The algorithm's effectiveness was confirmed in both stochastic and more realistic survival MAB problem settings.
• Optimal parameters for GWA-UCB1 and G-UCB1 were identified through preliminary investigations.

Conclusions:

• GWA-UCB1 offers a robust and effective solution for multi-armed bandit problems, outperforming established algorithms.
• The algorithm's simple modification of the UCB1 formula allows for easy integration into existing UCB-based reinforcement learning models.
• GWA-UCB1 is a valuable tool for various applications requiring efficient sequential decision-making under uncertainty.