通过偏差约束子集选择进行高效的基准测试
IEEE transactions on pattern analysis and machine intelligence
|August 12, 2025
概括
有效评估人工智能 (AI) 系统需要选择最佳基准子集. 这项研究引入了一个贪的算法,它保证了准确的AI模型得分估计,使用更少的数据,降低计算成本.
科学领域:
- 人工智能的人工智能
- 机器学习评估 机器学习评估
- 计算效率 计算效率 计算效率
背景情况:
- 评估大型人工智能 (AI) 模型是计算密集且昂贵的.
- 目前用于AI模型评估的方法通常需要广泛的基准,导致高资源支出.
- 需要有效的AI评估技术,尽量减少计算和人力成本.
研究的目的:
- 为了有效的AI模型评估,正式定义和近似选择子集问题.
- 开发一种方法来识别有价值的基准子集,以确保理论上的保证.
- 为了降低成本并提高评估大型AI模型的效率.
主要方法:
- 在高效的AI评估中对子集选择问题的正式定义和近似.
- 证明子集选择问题优化了一个子模块函数.
- 简单的贪算法的应用,用于统一的子集识别.
主要成果:
- 拟议的方法为人工智能得分估计中的偏差控制和概括性提供了第一个理论保证.
- 11个基准的语言模型的实验结果显示出卓越的性能.
- 精确的AI模型得分估计仅使用全基准的30%才能实现.
结论:
- 一个贪的算法有效地解决了有效的AI评估的子集选择问题.
- 这种方法显著减少了准确AI模型性能估计所需的数据.
- 这些发现促进了对人工智能系统的高效和稀疏的基准设计.
相关概念视频
Bias in Epidemiological Studies
676
Biases can arise at various stages of research, from study design and data collection to analysis and interpretation. Recognizing and addressing these biases is essential to ensure the validity and reliability of epidemiological findings.Broadly speaking, biases in epidemiology fall into three main categories: selection bias, information bias, and confounding. A more detailed description of possible biases is:
676
Bias
4.9K
Bias refers to any tendency that prevents a question from being considered unprejudiced. In research, bias occurs when one outcome or answer is selected or encouraged over others in sampling or testing. Bias can occur during any research phase, including study design, data collection, analysis, and publication.
In statistics, a sampling bias is created when a sample is collected from a population, and some members of the population are not as likely to be chosen as others (remember, each member...
In statistics, a sampling bias is created when a sample is collected from a population, and some members of the population are not as likely to be chosen as others (remember, each member...
4.9K
Quantifying and Rejecting Outliers: The Grubbs Test
2.0K
Sometimes, a data set can have a recorded numerical observation that greatly deviates from the rest of the data. Assuming that the data is normally distributed, a statistical method called the Grubbs test can be used to determine whether the observation is truly an outlier. To perform a two-tailed Grubbs test, first, calculate the absolute difference between the outlier and the mean. Then, calculate the ratio between this difference and the standard deviation of the sample. This...
2.0K
Friedman Two-way Analysis of Variance by Ranks
296
Friedman's Two-Way Analysis of Variance by Ranks is a nonparametric test designed to identify differences across multiple test attempts when traditional assumptions of normality and equal variances do not apply. Unlike conventional ANOVA, which requires normally distributed data with equal variances, Friedman's test is ideal for ordinal or non-normally distributed data, making it particularly useful for analyzing dependent samples, such as matched subjects over time or repeated measures...
296
Bonferroni Test
2.8K
The Bonferroni test is a statistical test named after Carlo Emilio Bonferroni, an Italian mathematician best known for Bonferroni inequalities. This statistical test is a type of multiple comparison test to determine which means are different than the rest. Bonferroni test can minimize the Type 1 error by reducing the significance level alpha, which otherwise increases with sample pairs.
The means of different samples are first paired in all possible combinations.
The null hypothesis of the...
The means of different samples are first paired in all possible combinations.
The null hypothesis of the...
2.8K
Detection of Gross Error: The Q Test
6.4K
When one or more data points appear far from the rest of the data, there is a need to determine whether they are outliers and whether they should be eliminated from the data set to ensure an accurate representation of the measured value. In many cases, outliers arise from gross errors (or human errors) and do not accurately reflect the underlying phenomenon. In some cases, however, these apparent outliers reflect true phenomenological differences. In these cases, we can use statistical methods...
6.4K


