缩小尺寸对异常值检测的影响:一个实证研究
Vivek Vaidya1, Jaideep Vaidya2
1East Brunswick High School, East Brunswick, NJ, USA.
概括
本研究探讨了异常值检测在较低维度中工作的效果,以及维度减小是否能够保持异常值的准确性. 了解这一点有助于为安全专家可视化复杂的数据.
科学领域:
- 数据科学数据科学数据科学
- 机器学习 机器学习
- 网络安全分析分析
背景情况:
- 异常值检测对于安全应用程序至关重要,但通常使用高维,复杂的数据.
- 对安全专家来说,鉴定异常值是具有挑战性的,因为在可视化高维空间方面存在困难.
研究的目的:
- 在较低的维度中调查异常值检测技术的有效性.
- 评估尺寸缩小方法如何保持异常值检测准确度.
- 为了确定数据可视化的程度,同时保持异常值的内在性质.
主要方法:
- 在缩小尺寸数据集上评估异常值检测性能.
- 将结果与传统的高维异常值检测进行比较.
- 分析缩小维度对异常值识别的影响.
主要成果:
- 异常值检测在较低维度中仍然有效.
- 缩小尺寸的技术可以保持异常值检测的准确性.
- 异常值的可视化得到了改进,但没有显著损失检测能力.
结论:
- 数据可以有效地可视化,以减少尺寸进行异常值分析.
- 减小维度是一种可行的方法,可以提高异常值检测在安全方面的可解释性.
- 这项研究弥合了复杂数据分析和安全环境中的人类理解之间的差距.
相关概念视频
Outliers and Influential Points
4.1K
An outlier is an observation of data that does not fit the rest of the data. It is sometimes called an extreme value. When you graph an outlier, it will appear not to fit the pattern of the graph. Some outliers are due to mistakes (for example, writing down 50 instead of 500), while others may indicate that something unusual is happening. Outliers are present far from the least squares line in the vertical direction. They have large "errors," where the "error" or residual is the...
4.1K
What Are Outliers?
3.9K
Outliers are observed data points that are far from the least squares line. They have unusual values and need to be examined carefully. Though an outlier may result from erroneous data, at other times, it may hold valuable information about the population under study and should be included in the data. Hence, it is crucial to examine what causes a data point to be an outlier.
The z score is used to find outliers or unusual values. It should be noted that any values beyond -2 and +2 are...
The z score is used to find outliers or unusual values. It should be noted that any values beyond -2 and +2 are...
3.9K
Detection of Gross Error: The Q Test
6.1K
When one or more data points appear far from the rest of the data, there is a need to determine whether they are outliers and whether they should be eliminated from the data set to ensure an accurate representation of the measured value. In many cases, outliers arise from gross errors (or human errors) and do not accurately reflect the underlying phenomenon. In some cases, however, these apparent outliers reflect true phenomenological differences. In these cases, we can use statistical methods...
6.1K
Quantifying and Rejecting Outliers: The Grubbs Test
1.6K
Sometimes, a data set can have a recorded numerical observation that greatly deviates from the rest of the data. Assuming that the data is normally distributed, a statistical method called the Grubbs test can be used to determine whether the observation is truly an outlier. To perform a two-tailed Grubbs test, first, calculate the absolute difference between the outlier and the mean. Then, calculate the ratio between this difference and the standard deviation of the sample. This...
1.6K
Modified Boxplots
9.7K
A standard box and whisker plot informs us about the spread of the data in a given sample. One can identify the minimum value, maximum value, first quartile value, second quartile or median value, and third quartile.
However, the box plot does not tell the reader about outliers - values that lie far from the center of the data. We can modify the standard box and whisker plot to identify the outliers and visualize the actual spread of the data in a sample.
Initially, we calculate the adjusted...
However, the box plot does not tell the reader about outliers - values that lie far from the center of the data. We can modify the standard box and whisker plot to identify the outliers and visualize the actual spread of the data in a sample.
Initially, we calculate the adjusted...
9.7K
Significance Testing: Overview
3.4K
Significance testing is a set of statistical methods used to test whether a claim about a parameter is valid. In analytical chemistry, significance testing is used primarily to determine whether the difference between two values comes from determinate or random errors. The effect of a particular change in the measurement protocol, analyst, or sample itself can cause a deviation from the expected result. In the case of a suspected deviation/outlier, we need to be able to confirm mathematically...
3.4K


