Q-ADER: An Effective Q-Learning for Recommendation With Diminishing Action Space

Area of Science:

• Artificial Intelligence
• Machine Learning
• Recommender Systems

Background:

• Deep reinforcement learning (RL) is popular for personalized recommender systems (PRSs) due to its ability to capture evolving user preferences.
• Deep Q-network (DQN) is a widely used RL technique in PRSs, known for its performance and simple updates.
• Many recommendation scenarios involve a diminishing action space to prevent recommending duplicate items.

Purpose of the Study:

• To address the action diminishing error in standard DQN algorithms when applied to diminishing action spaces in PRSs.
• To propose a novel algorithm that mitigates the discrepancy between fixed action spaces in Q-networks and dynamic recommendation environments.

Main Methods:

• Introduced the Q-learning-based action diminishing error reduction (Q-ADER) algorithm.
• Modified the temporal difference (TD) learning operator to incorporate an error reduction term.
• Augmented existing DQN algorithms with the Q-ADER error reduction component.

Main Results:

• Demonstrated that standard DQN methods are impractical for accurate value estimation in diminishing action spaces.
• Validated the effectiveness of the proposed Q-ADER algorithm through experiments on four real-world datasets.
• Showcased Q-ADER's ability to reduce action diminishing error and improve value estimates.

Conclusions:

• The action diminishing error poses a significant challenge for DQN-based PRSs in scenarios with decreasing action spaces.
• The Q-ADER algorithm offers a practical and effective solution to mitigate this error.
• Q-ADER enhances the performance of DQN algorithms in personalized recommendation tasks with diminishing action spaces.