在标题图像中寻找异常值的无监督方法
Jakub Zaprzałka1, Magdalena Topczewska1
1Faculty of Computer Science, Bialystok University of Technology, Wiejska 45A, 15-351 Bialystok, Poland.
Entropy (Basel, Switzerland)
|July 29, 2025
概括
在文本数据中识别异常记录对于公正的机器学习至关重要. 多维缩放和聚合技术可以有效地检测这些异常值,而米度-MDS和共弦距离显示出最佳性能.
科学领域:
- 机器学习 机器学习
- 数据科学数据科学数据科学
- 自然语言处理自然语言处理.
背景情况:
- 机器学习算法通常依赖于平均值,而平均值可能会被异常数据所偏差.
- 文本数据的高维度减少了距离区分,加剧了异常发现挑战.
- 现有的方法,如批量规范化,可以影响数据,需要替代解决方案.
研究的目的:
- 建议和评估用于检测基于文本的数据集中的异常记录的新方法.
- 为应对由于异常值导致的机器学习中偏差数据的挑战.
- 为了比较不同异常值检测技术的有效性.
主要方法:
- 多维缩放 (MDS) 具有公制和非公制版本.
- 使用最后的p和水平切断技术进行聚合集群.
- 与基于原始数据的方法进行比较,使用欧几里德和共弦距离.
主要成果:
- 直线距离被证明是检测异常值的最有效的度量.
- 基于metric-MDS的算法在人类评估的基础上表现出卓越的性能.
- 拟议的算法成功地识别了描述性图像字幕中的异常记录.
结论:
- 使用共弦距离的Metric-MDS是一种非常有效的方法,用于在文本数据中检测异常值.
- 准确识别异常值对于提高机器学习模型可靠性至关重要.
- 开发的技术为实现无偏见的数据集提供了强大的解决方案.
相关概念视频
What Are Outliers?
4.2K
Outliers are observed data points that are far from the least squares line. They have unusual values and need to be examined carefully. Though an outlier may result from erroneous data, at other times, it may hold valuable information about the population under study and should be included in the data. Hence, it is crucial to examine what causes a data point to be an outlier.
The z score is used to find outliers or unusual values. It should be noted that any values beyond -2 and +2 are...
The z score is used to find outliers or unusual values. It should be noted that any values beyond -2 and +2 are...
4.2K
Outliers and Influential Points
4.2K
An outlier is an observation of data that does not fit the rest of the data. It is sometimes called an extreme value. When you graph an outlier, it will appear not to fit the pattern of the graph. Some outliers are due to mistakes (for example, writing down 50 instead of 500), while others may indicate that something unusual is happening. Outliers are present far from the least squares line in the vertical direction. They have large "errors," where the "error" or residual is the...
4.2K
Quantifying and Rejecting Outliers: The Grubbs Test
2.1K
Sometimes, a data set can have a recorded numerical observation that greatly deviates from the rest of the data. Assuming that the data is normally distributed, a statistical method called the Grubbs test can be used to determine whether the observation is truly an outlier. To perform a two-tailed Grubbs test, first, calculate the absolute difference between the outlier and the mean. Then, calculate the ratio between this difference and the standard deviation of the sample. This...
2.1K
Detection of Gross Error: The Q Test
6.4K
When one or more data points appear far from the rest of the data, there is a need to determine whether they are outliers and whether they should be eliminated from the data set to ensure an accurate representation of the measured value. In many cases, outliers arise from gross errors (or human errors) and do not accurately reflect the underlying phenomenon. In some cases, however, these apparent outliers reflect true phenomenological differences. In these cases, we can use statistical methods...
6.4K
Modified Boxplots
10.1K
A standard box and whisker plot informs us about the spread of the data in a given sample. One can identify the minimum value, maximum value, first quartile value, second quartile or median value, and third quartile.
However, the box plot does not tell the reader about outliers - values that lie far from the center of the data. We can modify the standard box and whisker plot to identify the outliers and visualize the actual spread of the data in a sample.
Initially, we calculate the adjusted...
However, the box plot does not tell the reader about outliers - values that lie far from the center of the data. We can modify the standard box and whisker plot to identify the outliers and visualize the actual spread of the data in a sample.
Initially, we calculate the adjusted...
10.1K


