Benchmark suites instead of leaderboards for evaluating AI fairness

Area of Science:

• Artificial Intelligence
• Machine Learning Ethics
• AI Fairness

Background:

• Leaderboards and benchmarks are common tools for assessing artificial intelligence (AI) model fairness.
• Critics argue leaderboards incentivize optimizing for specific metrics, which is impossible due to varying application needs.
• Current critiques often entangle leaderboards and benchmarks, obscuring the value of benchmarks.

Purpose of the Study:

• To disentangle critiques of AI leaderboards and benchmarks.
• To propose a reformed approach to using benchmarks for AI fairness assessment.
• To advocate for the development of curated benchmark suites.

Main Methods:

• Analyzing the critiques of AI leaderboards and benchmarks.
• Conceptualizing the structure and purpose of benchmark suites.
• Outlining research directions for creating effective benchmark suites.

Main Results:

• Benchmarks, when separated from leaderboards, are valuable tools for understanding AI models.
• Curated benchmark suites can help researchers and practitioners identify a wide range of potential AI harms and fairness trade-offs.
• The proposed approach moves away from competitive leaderboards towards a more nuanced understanding of AI fairness.

Conclusions:

• Reformed benchmarks, organized into suites, offer a path to better monitor and improve AI fairness.
• Future research should focus on developing benchmark suites that address diverse usage, potential harms, and varied perspectives.
• Shifting focus from leaderboards to thoughtfully designed benchmark suites is crucial for advancing AI fairness.