Offline Evaluation of Recommender Systems in a User Interface With Multiple Carousels

Area of Science:

• Recommender Systems
• Human-Computer Interaction
• Information Retrieval

Background:

• Streaming services use multiple recommendation carousels (widgets) on user interfaces.
• Current offline evaluation methods fail to assess recommendation quality within a multi-carousel context.
• User interaction with recommendation carousels exhibits a 2D position bias, concentrating on the top-left.

Purpose of the Study:

• To propose an offline evaluation protocol for recommendation carousels that accounts for existing lists.
• To adapt ranking metrics for the 2D carousel interface, incorporating position bias.
• To develop and evaluate carousel ranking strategies for agnostic 2D layout generation.

Main Methods:

• Developed a novel offline evaluation protocol for carousel recommendation settings.
• Extended traditional ranking metrics to address the two-dimensional nature of carousel interfaces and position bias.
• Designed and tested two carousel ranking strategies for scenarios with layout-agnostic generation techniques.

Main Results:

• The proposed carousel evaluation protocol reveals significant shifts in algorithm ranking compared to traditional methods.
• Accounting for the 2D user interface layout and position bias leads to substantially different optimal carousel selections.
• Experiments on movie domain datasets demonstrate the practical impact of the new evaluation and ranking strategies.

Conclusions:

• A context-aware offline evaluation protocol is crucial for accurately assessing recommendation algorithms in carousel settings.
• Incorporating 2D position bias into evaluation and ranking is essential for optimizing user experience on streaming platforms.
• The proposed methods provide a more realistic assessment of recommendation performance and guide better UI design for carousels.