Hierarchical Multiagent Reinforcement Learning for Allocating Guaranteed Display Ads

Area of Science:

• Artificial Intelligence
• Computer Science
• Machine Learning

Background:

• Current methods for guaranteed display ads (GDAs) allocation struggle with dynamic, large-scale impression data and optimizing overall ad benefits.
• Existing approaches often assume static impressions or focus on individual ad performance, limiting their applicability in real-world advertising scenarios.

Purpose of the Study:

• To develop a novel method for the proactive allocation of display ads to impressions, ensuring contract demands are met in dynamic and large-scale advertising environments.
• To address the limitations of existing methods by optimizing the overall allocation of multiple GDAs simultaneously.

Main Methods:

• The problem is formulated as a sequential decision-making challenge within multi-agent reinforcement learning (MARL).
• A hierarchical MARL (HMARL) approach is proposed, featuring a manager policy and multiple subpolicies to handle numerous ads and impression dynamics.
• Each ad is assigned an allocation agent, and agents are coordinated to optimize GDA allocation based on ad states and impression data.

Main Results:

• HMARL demonstrated significant improvements over state-of-the-art approaches in extensive experiments.
• Experiments were conducted on three real-world datasets from the Tencent advertising platform, involving tens of millions of records.
• The proposed method effectively handles the complexities of dynamic impressions and a large number of ads.

Conclusions:

• The HMARL method provides an effective solution for optimizing guaranteed display ads allocation in large-scale, dynamic advertising systems.
• This research bridges the gap between theoretical GDA allocation problems and practical industrial production scenarios.
• The hierarchical structure of HMARL is crucial for managing agent policies and achieving superior performance in complex advertising environments.