发现大数据中的异常:一篇专注于应用元启发学和机器学习技术的综述
Claudia Cavallaro1, Vincenzo Cutello1, Mario Pavone1
1Department of Mathematics and Computer Science, University of Catania, Catania, Italy.
Frontiers in big data
|September 4, 2023
概括
在大数据中检测异常对于识别系统故障和网络攻击至关重要. 机器学习和深度学习方法提供了强大的解决方案,而元启发学提高了它们的效率.
科学领域:
- 计算机科学 计算机科学
- 数据科学数据科学数据科学
- 网络安全 网络安全
背景情况:
- 随着数据量不断增加,需要用于故障诊断和网络攻击识别的自动异常检测.
- 在大数据中手动标记异常值是不切实际的,因为规模.
- 异常检测产生警报,以防止系统故障和网络攻击,降低维护成本.
研究的目的:
- 提供最先进的异常检测方法的概述.
- 突出当前技术的优点,缺点和挑战.
- 探索元启发学在增强用于异常检测的机器学习中的作用.
主要方法:
- 审查机器学习和深度学习技术用于异常分类.
- 分析用于日志分析,过程分析和时间序列的方法.
- 探索与机器学习算法集成的元启发学.
主要成果:
- 没有单一的异常检测方法是普遍优越的;性能因数据集而异.
- 机器学习和深度学习是最近文学中占主导地位的方法.
- 超听觉学可以提高异常检测工具的稳定性和效率.
结论:
- 异常检测对于现代数据系统至关重要,特别是在网络安全方面.
- 选择异常检测方法的选择在很大程度上取决于特定的数据集.
- 整合元启发学为更有效的异常检测提供了一个有希望的方向.
相关概念视频
Heuristics
111
Heuristics are problem-solving strategies that use mental shortcuts to simplify decision-making. Unlike algorithms, which must be followed precisely to achieve a correct result, heuristics offer a general problem-solving framework. They save time and energy but can sometimes lead to less rational decisions.
People often rely on heuristics when faced with an overload of information, limited time, low importance of the decision, limited information, or when a heuristic readily comes to mind. For...
People often rely on heuristics when faced with an overload of information, limited time, low importance of the decision, limited information, or when a heuristic readily comes to mind. For...
111
Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving
79
Mechanistic models play a crucial role in algorithms for numerical problem-solving, particularly in nonlinear mixed effects modeling (NMEM). These models aim to minimize specific objective functions by evaluating various parameter estimates, leading to the development of systematic algorithms. In some cases, linearization techniques approximate the model using linear equations.
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
79
Statistical Inference Techniques in Hypothesis Testing: Parametric Versus Nonparametric Data
154
Statistical inference techniques, paramount in hypothesis testing, differentiate into two broad categories: parametric and nonparametric statistics.
Parametric statistics, as the name suggests, assumes that data follow a specific distribution, often a normal distribution. This assumption enables robust hypothesis testing and estimation. Parametric methods, like the Student's t-test or Goodness-of-fit test, are frequently employed in biostatistics due to their robustness. For instance,...
Parametric statistics, as the name suggests, assumes that data follow a specific distribution, often a normal distribution. This assumption enables robust hypothesis testing and estimation. Parametric methods, like the Student's t-test or Goodness-of-fit test, are frequently employed in biostatistics due to their robustness. For instance,...
154
Steps in Outbreak Investigation
152
In the ever-evolving field of public health, statistical analysis serves as a cornerstone for understanding and managing disease outbreaks. By leveraging various statistical tools, health professionals can predict potential outbreaks, analyze ongoing situations, and devise effective responses to mitigate impact. For that to happen, there are a few possible stages of the analysis:
152
Detection of Gross Error: The Q Test
6.2K
When one or more data points appear far from the rest of the data, there is a need to determine whether they are outliers and whether they should be eliminated from the data set to ensure an accurate representation of the measured value. In many cases, outliers arise from gross errors (or human errors) and do not accurately reflect the underlying phenomenon. In some cases, however, these apparent outliers reflect true phenomenological differences. In these cases, we can use statistical methods...
6.2K


