Efficient Benchmarking via Bias-Bounded Subset Selection.

Area of Science:

• Artificial Intelligence
• Machine Learning Evaluation
• Computational Efficiency

Background:

• Evaluating large artificial intelligence (AI) models is computationally intensive and costly.
• Current methods for AI model evaluation often require extensive benchmarks, leading to high resource expenditure.
• There is a need for efficient AI evaluation techniques that minimize computational and human costs.

Purpose of the Study:

• To formally define and approximate the subset selection problem for efficient AI model evaluation.
• To develop a method for identifying valuable benchmark subsets that ensures theoretical guarantees.
• To reduce the cost and improve the efficiency of evaluating large AI models.

Main Methods:

• Formal definition and approximation of the subset selection problem in efficient AI evaluation.
• Proof that the subset selection problem optimizes a submodular function.
• Application of a simple greedy algorithm for unified subset identification.

Main Results:

• The proposed method provides the first theoretical guarantees for bias control and generalizability in AI score estimation.
• Experimental results with language models across 11 benchmarks demonstrate superior performance.
• Accurate AI model score estimation achieved using only 30% of the full benchmark.

Conclusions:

• A greedy algorithm effectively solves the subset selection problem for efficient AI evaluation.
• This approach significantly reduces the data required for accurate AI model performance estimation.
• The findings facilitate efficient and sparse benchmark design for AI systems.