对大数据设置中的对应概率的计算高效近似
Robin Van Oirbeek1, Jolien Ponnet2, Bart Baesens3,4
1Data Office, Allianz Benelux, Brussels, Belgium.
Big data
|June 8, 2023
概括
我们开发了两种快速而准确的估计方法来计算对应概率,这是统计模型性能的一个关键指标. 这些方法适用于离散数据和连续数据,大大减少了大型数据集的计算时间.
科学领域:
- 统计 统计 统计 统计
- 机器学习 机器学习
- 数据科学数据科学数据科学
背景情况:
- 性能测量对于统计模型至关重要.
- 曲线下的面积 (AUC) 是一个流行的二进制分类指标,相当于对应概率.
- 一致概率扩展到连续响应变量,与AUC不同.
研究的目的:
- 提出两个新的,快速的,准确的估计方法,用于对应概率.
- 解决在大型数据集中计算歧视性措施的计算挑战.
- 提供适用于离散和连续响应变量的方法.
主要方法:
- 开发两个新的对一致概率的估计算法.
- 将方法应用于离散和连续数据设置.
- 通过广泛的模拟研究和现实数据集进行验证.
主要成果:
- 建议的估计器表现出色的表现.
- 与传统方法相比,这两种方法都实现了显著更快的计算时间.
- 模拟研究证实了估计器的准确性和效率.
结论:
- 新的估计方法为计算一致性概率提供了有效的解决方案.
- 这些方法适用于大规模的统计建模和绩效评估.
- 这些发现在模拟和真实世界的数据场景中都很强大.
更多相关视频
相关概念视频
Kendall's Coefficient of Concordance
432
Kendall's Coefficient of Concordance (W), also known as Kendall's W, is a non-parametric statistical measure used to assess the agreement or concordance between multiple raters or judges when they rank a set of items. It is often used when you have ordinal data (ranks) and you want to see if there is consistency or consensus among the raters. It is widely applied in research areas such as psychology, medicine, and social sciences, where multiple judges are asked to rank or rate subjects...
432
Expected Frequencies in Goodness-of-Fit Tests
2.6K
A goodness-of-fit test is conducted to determine whether the observed frequency values are statistically similar to the frequencies expected for the dataset. Suppose the expected frequencies for a dataset are equal such as when predicting the frequency of any number appearing when casting a die. In that case, the expected frequency is the ratio of the total number of observations (n) to the number of categories (k).
2.6K
Accuracy, limits, and approximation
484
Accuracy, limits, and approximations are common in many fields, especially in engineering calculations. These concepts are imperative for ensuring that a given value is as close as possible to its true value.
Accuracy is defined as the closeness of the measured value to the true or actual value. In engineering mechanics, repeated measurements are taken during theoretical or experimental analyses to ensure that the result is precise and accurate.
The accuracy of any solution is based on the...
Accuracy is defined as the closeness of the measured value to the true or actual value. In engineering mechanics, repeated measurements are taken during theoretical or experimental analyses to ensure that the result is precise and accurate.
The accuracy of any solution is based on the...
484
Testing a Claim about Population Proportion
3.4K
A complete procedure for testing a claim about a population proportion is provided here.
There are two methods of testing a claim about a population proportion: (1) Using the sample proportion from the data where a binomial distribution is approximated to the normal distribution and (2) Using the binomial probabilities calculated from the data.
The first method uses normal distribution as an approximation to the binomial distribution. The requirements are as follows: sample size is large...
There are two methods of testing a claim about a population proportion: (1) Using the sample proportion from the data where a binomial distribution is approximated to the normal distribution and (2) Using the binomial probabilities calculated from the data.
The first method uses normal distribution as an approximation to the binomial distribution. The requirements are as follows: sample size is large...
3.4K
Probability in Statistics
13.5K
Probability is the likelihood of an event occurring. The term event is defined as a collection of results of a procedure. An event is a simple event when an outcome cannot be divided into simpler parts.
An example of a simple event is a coin toss. The result of a coin toss is either a head or a tail. Here, head and tail are two simple events. These two simple events make up the sample space. Further, the probability of an event occurring falls within the range of 0 to 1. The probability of an...
An example of a simple event is a coin toss. The result of a coin toss is either a head or a tail. Here, head and tail are two simple events. These two simple events make up the sample space. Further, the probability of an event occurring falls within the range of 0 to 1. The probability of an...
13.5K
Fisher's Exact Test
653
Fisher's exact test is a statistical significance test widely used to analyze 2x2 contingency tables, particularly in situations where sample sizes are small. Unlike the chi-squared test, which approximates P-values and assumes minimum expected frequencies of at least five in each cell, Fisher's exact test calculates the exact probability (P-value) of observing the data or more extreme results under the null hypothesis. This feature makes it especially valuable when the assumptions of...
653


