ConAnomaly: Content-Based Anomaly Detection for System Logs.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Enterprise systems generate extensive logs crucial for system maintenance and business management.
• Current log anomaly detection methods often rely on log parsers and struggle with unknown log types and semantic nuances.
• Existing approaches fail to fully leverage the rich semantic information present within log data.

Purpose of the Study:

• To introduce ConAnomaly, a novel log-based anomaly detection model.
• To address the limitations of existing methods in handling unknown log types and capturing log semantics.
• To improve the accuracy and stability of log anomaly detection.

Main Methods:

• Developed ConAnomaly, integrating a log sequence encoder (log2vec) with a multi-layer Long Short-Term Memory (LSTM) network.
• Designed log2vec using Word2vec to vectorize log content, filter invalid words via part-of-speech tagging, and derive sequence vectors.
• Employed weighted averaging to obtain sequence vectors, capturing both semantic and sequential log information.

Main Results:

• ConAnomaly demonstrated good stability across evaluations on two log datasets.
• The model showed capability in handling previously unseen log types.
• Experimental results indicated superior performance compared to most existing log-based anomaly detection methods.

Conclusions:

• ConAnomaly effectively utilizes log semantic information and sequential relationships for anomaly detection.
• The proposed model offers improved robustness and performance, particularly with diverse and unknown log types.
• ConAnomaly represents a significant advancement in log anomaly detection techniques.