EAD:轻松检测异常,是一种基于深度学习的方法,用于检测英语文本数据中的异常值
1School of Foreign Languages, Zhengzhou College of Finance and Economics, Zhengzhou, Henan, China.
PeerJ. Computer science
|December 9, 2024
概括
本研究介绍了一种无监督的系统,使用中心体嵌入和最小共差决定器 (MCD) 来有效地检测大文本数据集中的异常. 这种新的方法实现了强大的准确性,改善了决策和欺诈检测流程.
科学领域:
- 数据科学数据科学数据科学
- 机器学习 机器学习
- 自然语言处理自然语言处理.
背景情况:
- 异常检测对于决策,欺诈检测和自动分类至关重要.
- 现有的异常检测模型通常依赖于大型数据集的计算成本昂贵的传统代币化方法.
- 在广泛的文本数据中识别异常值会带来重大的计算挑战.
研究的目的:
- 提出一个无监督的系统来检测大型文本卷中的异常值.
- 开发一个计算效率高的异常检测方法.
- 提高识别异常的准确性,同时将其与新数据区分开来.
主要方法:
- 使用完全基于MiniLM-L6-v2的系统进行异常检测.
- 采用中枢嵌入式用于在高多样性,大容量数据中提取异常值.
- 纳入基于最小协差决定因素 (MCD) 的方法来区分新性和异常值.
主要成果:
- 在各种数据集上实现了94%的精确度和0.95的F1得分.
- 在相对较大的脚本中展示了有效的异常追踪能力.
- 提出的方法,作为应用程序实施. 在Python中用于异常检测 (AAD),显示出强大的性能.
结论:
- 拟议的无监督系统有效地检测大型文本数据中的异常.
- 该方法为传统的代币化技术提供了一个计算效率高的替代方案.
- 适用于需要文字数据分析的领域,如人力资源管理和安全.
相关概念视频
Outliers and Influential Points
4.0K
An outlier is an observation of data that does not fit the rest of the data. It is sometimes called an extreme value. When you graph an outlier, it will appear not to fit the pattern of the graph. Some outliers are due to mistakes (for example, writing down 50 instead of 500), while others may indicate that something unusual is happening. Outliers are present far from the least squares line in the vertical direction. They have large "errors," where the "error" or residual is the...
4.0K
What Are Outliers?
3.6K
Outliers are observed data points that are far from the least squares line. They have unusual values and need to be examined carefully. Though an outlier may result from erroneous data, at other times, it may hold valuable information about the population under study and should be included in the data. Hence, it is crucial to examine what causes a data point to be an outlier.
The z score is used to find outliers or unusual values. It should be noted that any values beyond -2 and +2 are...
The z score is used to find outliers or unusual values. It should be noted that any values beyond -2 and +2 are...
3.6K
Quantifying and Rejecting Outliers: The Grubbs Test
1.5K
Sometimes, a data set can have a recorded numerical observation that greatly deviates from the rest of the data. Assuming that the data is normally distributed, a statistical method called the Grubbs test can be used to determine whether the observation is truly an outlier. To perform a two-tailed Grubbs test, first, calculate the absolute difference between the outlier and the mean. Then, calculate the ratio between this difference and the standard deviation of the sample. This...
1.5K
Detection of Gross Error: The Q Test
5.6K
When one or more data points appear far from the rest of the data, there is a need to determine whether they are outliers and whether they should be eliminated from the data set to ensure an accurate representation of the measured value. In many cases, outliers arise from gross errors (or human errors) and do not accurately reflect the underlying phenomenon. In some cases, however, these apparent outliers reflect true phenomenological differences. In these cases, we can use statistical methods...
5.6K
Types of Errors: Detection and Minimization
1.4K
Error is the deviation of the obtained result from the true, expected value or the estimated central value. Errors are expressed in absolute or relative terms.
Absolute error in a measurement is the numerical difference from the true or central value. Relative error is the ratio between absolute error and the true or central value, expressed as a percentage.
Errors can be classified by source, magnitude, and sign. There are three types of errors: systematic, random, and gross.
Systematic or...
Absolute error in a measurement is the numerical difference from the true or central value. Relative error is the ratio between absolute error and the true or central value, expressed as a percentage.
Errors can be classified by source, magnitude, and sign. There are three types of errors: systematic, random, and gross.
Systematic or...
1.4K
Random Error
821
Random or indeterminate errors originate from various uncontrollable variables, such as variations in environmental conditions, instrument imperfections, or the inherent variability of the phenomena being measured. Usually, these errors cannot be predicted, estimated, or characterized because their direction and magnitude often vary in magnitude and direction even during consecutive measurements. As a result, they are difficult to eliminate. However, the aggregate effect of these errors can be...
821


