基于对比学习和检索增强的系统日志异常检测
Weian Li1, Yang Wu1, Wei Huang2
1School of Big Data and Computer Science, Guizhou Normal University, 550025, Guiyang, China.
Scientific reports
|November 3, 2025
概括
一个新的框架LogSentry,通过使用对比学习和检索增强方法来增强日志异常检测. 这种方法有效地解决了日志变化和改进系统监控的新格式等挑战.
科学领域:
- 计算机科学 计算机科学
- 人工智能的人工智能
- 网络安全 网络安全
背景情况:
- 系统日志记录关键运行时事件,对于快速发现问题至关重要.
- 记录异常检测至关重要,但由于日志变化,数据不平衡和不断变化的格式而受到挑战.
研究的目的:
- 提出LogSentry,这是一个强大的日志异常检测框架.
- 通过使用先进的机器学习技术,克服日志异常检测的现有挑战.
主要方法:
- 一个基于BERT的模型,用于预训练和微调的对比学习.
- 一个采集增强的推断阶段使用K-最近邻居 (KNN).
- 模型预测和检索结果的加权总和,用于最终的异常分类.
主要成果:
- 在广泛使用的日志数据集上,LogSentry框架表现出高性能.
- 与现有的基线方法相比,在日志异常检测方面取得了优异的结果.
结论:
- LogSentry有效地解决了日志异常检测中的关键挑战.
- 拟议的框架为可靠的系统日志分析和安全提供了一个有希望的解决方案.
相关概念视频
Types of Errors: Detection and Minimization
Error is the deviation of the obtained result from the true, expected value or the estimated central value. Errors are expressed in absolute or relative terms.
Absolute error in a measurement is the numerical difference from the true or central value. Relative error is the ratio between absolute error and the true or central value, expressed as a percentage.
Errors can be classified by source, magnitude, and sign. There are three types of errors: systematic, random, and gross.
Systematic or...
Absolute error in a measurement is the numerical difference from the true or central value. Relative error is the ratio between absolute error and the true or central value, expressed as a percentage.
Errors can be classified by source, magnitude, and sign. There are three types of errors: systematic, random, and gross.
Systematic or...
Systematic Error: Methodological and Sampling Errors
In the case of systematic errors, the sources can be identified, and the errors can be subsequently minimized by addressing these sources. According to the source, systematic errors can be divided into sampling, instrumental, methodological, and personal errors.
Sampling errors originate from improper sampling methods or the wrong sample population. These errors can be minimized by refining the sampling strategy. Defective instruments or faulty calibrations are the sources of instrumental...
Sampling errors originate from improper sampling methods or the wrong sample population. These errors can be minimized by refining the sampling strategy. Defective instruments or faulty calibrations are the sources of instrumental...
Detection of Gross Error: The Q Test
When one or more data points appear far from the rest of the data, there is a need to determine whether they are outliers and whether they should be eliminated from the data set to ensure an accurate representation of the measured value. In many cases, outliers arise from gross errors (or human errors) and do not accurately reflect the underlying phenomenon. In some cases, however, these apparent outliers reflect true phenomenological differences. In these cases, we can use statistical methods...
Difference from Background: Limit of Detection
The limit of detection (LOD) is the smallest amount of analyte that can be distinguished from the background noise. The LOD value corresponds to the concentration at which the analyte signal is three times larger than the standard deviation of the blank signal. Below this value, the analyte signal cannot be differentiated from the background noise. It is calculated by dividing the calibration slope by 3 times the standard deviation of the blank signals.
The LOD indicates the presence or absence...
The LOD indicates the presence or absence...
Mass Analyzers: Overview
The mass analyzer is a crucial component of the mass spectrometer. In the ionization chamber, the vaporized sample is bombarded with a high-energy electron beam to generate a radical cation and further fragment into neutral molecules, radicals, and cations. A series of negatively charged accelerator plates accelerate the cations into the mass analyzer. The mass analyzer separates ions according to their mass-to-charge (m/z) ratios and then directs them to the detector. The common types of mass...
