DaisyRec 2.0: Benchmarking Recommendation for Rigorous Evaluation.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Recommender systems lack standardized evaluation benchmarks, leading to unreproducible results and unfair comparisons.
• Existing evaluation methods are inconsistent, impacting the reliability and comparability of research findings.

Purpose of the Study:

• To address the critical issue of inadequate benchmarks in recommender system evaluation.
• To propose standardized procedures and provide empirical evidence for rigorous recommendation evaluation.

Main Methods:

• Conducted a systematic review of 141 papers (2017-2020) to identify hyper-factors influencing recommendation performance.
• Classified hyper-factors into model-independent and model-dependent categories.
• Developed and released the DaisyRec 2.0 library for integrating hyper-factors and performing empirical studies.

Main Results:

• Identified and analyzed key hyper-factors affecting recommendation evaluation.
• Demonstrated the impact of different hyper-factors on recommendation performance through a holistic empirical study.
• Established standardized procedures for rigorous evaluation and provided baseline performance metrics.

Conclusions:

• The proposed benchmarks and standardized procedures enhance the rigor, reproducibility, and fairness of recommender system evaluation.
• The DaisyRec 2.0 library serves as a valuable tool for researchers to conduct consistent and reliable evaluations.
• This work lays the foundation for future research in developing more robust and trustworthy recommender systems.