基于特征的规范模型用于异常检测
Hui Yie Teh1, Kevin I-Kai Wang1, Andreas W Kempa-Liehr2
1Department of Electrical, Computer and Software Engineering, The University of Auckland, Auckland 1142, New Zealand.
Sensors (Basel, Switzerland)
|August 14, 2025
概括
本研究介绍了物联网传感器数据的异常检测框架. 它有效地从有限的数据中学习传感器特定的正常性模型,提高异常检测的准确性.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 传感器数据分析数据分析
背景情况:
- 由于传感器特定的特性和短的校准周期,检测传感器数据中的新奇异常是具有挑战性的.
- 物联网 (IoT) 部署中的低成本传感器通常具有较低的数据质量,需要改进异常检测方法.
- 现有的方法与传感器数据的异质性以及从有限的数据中快速学习的需要作斗争.
研究的目的:
- 开发一种异常检测框架,能够从有限的,无异常数据中学习传感器特定的正常模型.
- 为应对从物联网部署中来自异质传感器数据中检测以前未见的异常的挑战.
- 在资源有限的物联网环境中提高异常检测的准确性和可靠性.
主要方法:
- 一个框架,学习个别传感器特定的正常模型使用无监督的功能工程.
- 使用在统计学上显著特征上训练的局部异常因子 (LOF) 模型来识别异常.
- 对现实环境监测数据集进行评估,校准时间极短 (例如3天或10%的数据).
主要成果:
- 与四种最先进的异常检测方法相比,拟议的框架实现了更高的性能.
- 在F1得分的改善范围从5.4%到9.3%,而马修斯相关系数提高了4.0%到7.6%.
- 即使使用非常有限的校准数据,也证明了有效的异常检测,突出显示了框架的效率.
结论:
- 开发的框架成功地学习了传感器特定的正常性模型,从而在物联网设置中实现了强大的异常检测.
- 该方法在处理异质传感器数据和短校准周期方面是有效的,性能优于现有方法.
- 这项工作为提高物联网应用中低成本传感器数据可靠性提供了有价值的工具.
相关概念视频
Quantifying and Rejecting Outliers: The Grubbs Test
2.0K
Sometimes, a data set can have a recorded numerical observation that greatly deviates from the rest of the data. Assuming that the data is normally distributed, a statistical method called the Grubbs test can be used to determine whether the observation is truly an outlier. To perform a two-tailed Grubbs test, first, calculate the absolute difference between the outlier and the mean. Then, calculate the ratio between this difference and the standard deviation of the sample. This...
2.0K
Expected Frequencies in Goodness-of-Fit Tests
2.6K
A goodness-of-fit test is conducted to determine whether the observed frequency values are statistically similar to the frequencies expected for the dataset. Suppose the expected frequencies for a dataset are equal such as when predicting the frequency of any number appearing when casting a die. In that case, the expected frequency is the ratio of the total number of observations (n) to the number of categories (k).
2.6K
Detection of Gross Error: The Q Test
6.4K
When one or more data points appear far from the rest of the data, there is a need to determine whether they are outliers and whether they should be eliminated from the data set to ensure an accurate representation of the measured value. In many cases, outliers arise from gross errors (or human errors) and do not accurately reflect the underlying phenomenon. In some cases, however, these apparent outliers reflect true phenomenological differences. In these cases, we can use statistical methods...
6.4K
The Anderson-Darling Test
876
The Anderson-Darling test is a statistical method used to determine whether a data sample is likely drawn from a specific theoretical distribution. Unlike parametric tests, it does not require assumptions about specific parameters of the distribution. Instead, it compares the sample's empirical cumulative distribution function (ECDF) with the cumulative distribution function (CDF) of the hypothesized distribution. Critical values for the test are specific to the chosen distribution rather...
876
What Are Outliers?
4.2K
Outliers are observed data points that are far from the least squares line. They have unusual values and need to be examined carefully. Though an outlier may result from erroneous data, at other times, it may hold valuable information about the population under study and should be included in the data. Hence, it is crucial to examine what causes a data point to be an outlier.
The z score is used to find outliers or unusual values. It should be noted that any values beyond -2 and +2 are...
The z score is used to find outliers or unusual values. It should be noted that any values beyond -2 and +2 are...
4.2K
Statistical Inference Techniques in Hypothesis Testing: Parametric Versus Nonparametric Data
209
Statistical inference techniques, paramount in hypothesis testing, differentiate into two broad categories: parametric and nonparametric statistics.
Parametric statistics, as the name suggests, assumes that data follow a specific distribution, often a normal distribution. This assumption enables robust hypothesis testing and estimation. Parametric methods, like the Student's t-test or Goodness-of-fit test, are frequently employed in biostatistics due to their robustness. For instance,...
Parametric statistics, as the name suggests, assumes that data follow a specific distribution, often a normal distribution. This assumption enables robust hypothesis testing and estimation. Parametric methods, like the Student's t-test or Goodness-of-fit test, are frequently employed in biostatistics due to their robustness. For instance,...
209


