走向最佳模型评估:通过积极改进的估计器增强主动测试
JooChul Lee1, Likhitha Kolla2, Jinbo Chen2
1Department of Biostatistics, Epidemiology and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, 19104, USA. joochul.lee@pennmedicine.upenn.edu.
Scientific reports
|May 9, 2024
概括
本研究引入了新的主动测试 (AT) 方法,用于评估具有较少标记数据的机器学习模型. 拟议的估计器,AILUR和AIIPW,降低标签成本并提高模型可靠性.
科学领域:
- 机器学习 机器学习
- 统计建模 统计建模
- 数据科学数据科学数据科学
背景情况:
- 评估大型机器学习模型需要大量的标记数据,这正变得不切实际.
- 传统的模型评估方法面临着数据集规模不断增加的挑战.
- 主动测试 (AT) 通过选择性抽样数据进行评估提供了一个解决方案.
研究的目的:
- 为积极测试开发新型估计器,以提高模型评估效率.
- 减少对完全标记数据集的依赖,以评估预先训练的模型.
- 提高机器学习中模型评估的成本效益.
主要方法:
- 提出了两个新型估计器:积极改进的水平化公正风险 (AILUR) 和积极改进的反向概率权重 (AIIPW).
- 来自非参数光滑估计技术的推导估计.
- 引入了AIIPW估计器的模型重新校准过程,以优化AT框架内的抽样概率.
主要成果:
- 拟议的AILUR和AIIPW估计器在四个现实数据集上始终超过现有的活跃测试方法.
- 在不同的亚样本大小中证明了新方法的稳定性.
- 展示了拟议方法在显著降低标签成本方面的有效性.
结论:
- 新的AILUR和AIIPW估计器为评估机器学习模型提供了更有效和更具成本效益的方法.
- 这些积极的测试方法对于大数据时代可靠的模型评估至关重要.
- 开发的技术提供了切实可行的解决方案,可以降低标签费用,同时保持评估准确性.
相关概念视频
Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving
51
Mechanistic models play a crucial role in algorithms for numerical problem-solving, particularly in nonlinear mixed effects modeling (NMEM). These models aim to minimize specific objective functions by evaluating various parameter estimates, leading to the development of systematic algorithms. In some cases, linearization techniques approximate the model using linear equations.
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
51
One-Compartment Open Model: Wagner-Nelson and Loo Riegelman Method for ka Estimation
482
This lesson introduces two critical methods in pharmacokinetics, the Wagner-Nelson and Loo-Riegelman methods, used for estimating the absorption rate constant (ka) for drugs administered via non-intravenous routes. The Wagner-Nelson method relates ka to the plasma concentration derived from the slope of a semilog percent unabsorbed time plot. However, it is limited to drugs with one-compartment kinetics and can be impacted by factors like gastrointestinal motility or enzymatic degradation.
On...
On...
482
Goodness-of-Fit Test
3.3K
The goodness-of-fit test is a type of hypothesis test which determines whether the data "fits" a particular distribution. For example, one may suspect that some anonymous data may fit a binomial distribution. A chi-square test (meaning the distribution for the hypothesis test is chi-square) can be used to determine if there is a fit. The null and alternative hypotheses may be written in sentences or stated as equations or inequalities. The test statistic for a goodness-of-fit test is given as...
3.3K
Testing a Claim about Standard Deviation
2.4K
A complete procedure to test a claim about population standard deviation or population variance is explained here.
The hypothesis testing for the claim of population standard deviation (or variance) requires the data and samples to be random and unbiased. The population distribution also must be normal. There is no specific requirement on the sample size as the estimation is based on the chi-square distribution.
As a first step, the hypothesis (null and alternative) concerning the claim about...
The hypothesis testing for the claim of population standard deviation (or variance) requires the data and samples to be random and unbiased. The population distribution also must be normal. There is no specific requirement on the sample size as the estimation is based on the chi-square distribution.
As a first step, the hypothesis (null and alternative) concerning the claim about...
2.4K
Expected Frequencies in Goodness-of-Fit Tests
2.5K
A goodness-of-fit test is conducted to determine whether the observed frequency values are statistically similar to the frequencies expected for the dataset. Suppose the expected frequencies for a dataset are equal such as when predicting the frequency of any number appearing when casting a die. In that case, the expected frequency is the ratio of the total number of observations (n) to the number of categories (k).
2.5K
Accuracy and Errors in Hypothesis Testing
198
Hypothesis testing is a fundamental statistical tool that begins with the assumption that the null hypothesis H0 is true. During this process, two types of errors can occur: Type I and Type II. A Type I error refers to the incorrect rejection of a true null hypothesis, while a Type II error involves the failure to reject a false null hypothesis.
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...
198


