A sampling approach to Debiasing the offline evaluation of recommender systems

Area of Science:

• Computer Science
• Machine Learning
• Data Science

Background:

• Offline evaluation of recommender systems (RSs) heavily relies on historical data, which is frequently biased.
• This bias arises from confounders in data collection, leading to user-item interactions being Missing Not At Random (MNAR).
• Performance metrics on MNAR data may not accurately reflect real-world RS performance without bias mitigation.

Purpose of the Study:

• To investigate a sampling approach as an alternative to biased offline evaluation metrics for recommender systems.
• To propose and formulate novel sampling strategies, WTD and WTD_H, to mitigate bias in MNAR data.
• To empirically validate the effectiveness of existing and proposed sampling methods in estimating unbiased recommender system performance.

Main Methods:

• A comprehensive literature survey on bias in RS offline evaluation.
• Formulation of two novel sampling approaches: WTD and a practical variant WTD_H.
• Comparison of WTD/WTD_H against the SKEW sampling strategy and random intervention baselines on MNAR data.

Main Results:

• Empirical validation of the SKEW sampling strategy's effectiveness for the first time.
• Demonstration that the proposed WTD and WTD_H approaches provide better estimates of performance on unbiased Missing At Random (MAR) test data.
• The WTD strategy exhibits high generality and low overhead, applicable to training and evaluation without requiring learning.

Conclusions:

• Sampling strategies offer a viable alternative to developing new metrics for addressing bias in recommender system offline evaluation.
• The proposed WTD and WTD_H methods effectively generate less biased test sets, leading to more reliable performance estimations.
• The WTD approach is a general, low-overhead solution that improves the accuracy of offline evaluations for recommender systems.