STAR_outliers:一个python包,将单变异的异常值与非正常分布分开
1Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania, Philadelphia, PA, USA.
BioData mining
|September 4, 2023
概括
STAR_outliers有效地从各种数据分布中删除单变异的异常值,在模拟和真实数据的精度和回忆方面超过现有方法. 这种新的算法准确地模拟了各种分布形状,提高了异常值检测的准确性.
科学领域:
- 统计 统计 统计 统计
- 数据科学数据科学数据科学
- 机器学习 机器学习
背景情况:
- 对于任意形状的分布,单变异异常值的检测具有挑战性.
- 现有的方法在扭曲,曲,双模式和单调性方面扎.
- 多变量算法通常不适合单变量异常值去除.
研究的目的:
- 为各种数据分布开发一个强大的单变异异常检测算法.
- 为了解决模拟复杂分布形状的当前方法的局限性.
- 为了提高异常点移除的准确性和可靠性.
主要方法:
- 实现了Skew和Tail重度调整的异常值 (STAR_outliers) 算法.
- 结合已建立的算法来建模任意形状的单变量分布.
- 根据正常性假设方法,隔离森林和基于IQR的算法验证的STAR_异常值.
主要成果:
- STAR_异常值在消除模拟异常值方面表现出卓越的回忆力和精度.
- 该算法准确地模拟了现实世界NHANES数据中的异常值边界.
- 与其他方法相比,STAR_outliers删除了更准确的百分比值 (0.7%).
结论:
- STAR_outliers是一个灵活而有效的Python包,用于删除单变异异常值.
- 该算法在各种分布类型中表现优于常见的异常值检测方法.
- STAR_outliers为真实世界数据集提供了更好的准确性.
相关概念视频
What Are Outliers?
3.9K
Outliers are observed data points that are far from the least squares line. They have unusual values and need to be examined carefully. Though an outlier may result from erroneous data, at other times, it may hold valuable information about the population under study and should be included in the data. Hence, it is crucial to examine what causes a data point to be an outlier.
The z score is used to find outliers or unusual values. It should be noted that any values beyond -2 and +2 are...
The z score is used to find outliers or unusual values. It should be noted that any values beyond -2 and +2 are...
3.9K
Outliers and Influential Points
4.1K
An outlier is an observation of data that does not fit the rest of the data. It is sometimes called an extreme value. When you graph an outlier, it will appear not to fit the pattern of the graph. Some outliers are due to mistakes (for example, writing down 50 instead of 500), while others may indicate that something unusual is happening. Outliers are present far from the least squares line in the vertical direction. They have large "errors," where the "error" or residual is the...
4.1K
Quantifying and Rejecting Outliers: The Grubbs Test
1.6K
Sometimes, a data set can have a recorded numerical observation that greatly deviates from the rest of the data. Assuming that the data is normally distributed, a statistical method called the Grubbs test can be used to determine whether the observation is truly an outlier. To perform a two-tailed Grubbs test, first, calculate the absolute difference between the outlier and the mean. Then, calculate the ratio between this difference and the standard deviation of the sample. This...
1.6K
Modified Boxplots
9.8K
A standard box and whisker plot informs us about the spread of the data in a given sample. One can identify the minimum value, maximum value, first quartile value, second quartile or median value, and third quartile.
However, the box plot does not tell the reader about outliers - values that lie far from the center of the data. We can modify the standard box and whisker plot to identify the outliers and visualize the actual spread of the data in a sample.
Initially, we calculate the adjusted...
However, the box plot does not tell the reader about outliers - values that lie far from the center of the data. We can modify the standard box and whisker plot to identify the outliers and visualize the actual spread of the data in a sample.
Initially, we calculate the adjusted...
9.8K
Detection of Gross Error: The Q Test
6.2K
When one or more data points appear far from the rest of the data, there is a need to determine whether they are outliers and whether they should be eliminated from the data set to ensure an accurate representation of the measured value. In many cases, outliers arise from gross errors (or human errors) and do not accurately reflect the underlying phenomenon. In some cases, however, these apparent outliers reflect true phenomenological differences. In these cases, we can use statistical methods...
6.2K
Unusual Results
3.2K
Unusual results are those that have a very low chance of occurring. Unusual results can be identified using probabilities and the range rule of thumb. In problems involving probability, unusual results can be observed in 2 instances – an unusually high number of successes or an unusually low number of successes.
According to the range rule of thumb, any value above or below two standard deviations, 2σ from the mean, μ is considered unusual.
Maximum unusual value =...
According to the range rule of thumb, any value above or below two standard deviations, 2σ from the mean, μ is considered unusual.
Maximum unusual value =...
3.2K


