Two Class Pruned Log Message Anomaly Detection.

Area of Science:

• Computer Science
• Machine Learning
• Data Mining

Background:

• Log messages are crucial for system monitoring and anomaly detection.
• Analyzing unstructured log data presents significant challenges due to its complexity and volume.
• Existing anomaly detection methods struggle with the nuances of log message interpretation.

Purpose of the Study:

• To propose a hybrid log message anomaly detection technique.
• To improve the accuracy and reliability of anomaly detection in system logs.
• To leverage deep learning for enhanced log analysis.

Main Methods:

• A hybrid approach involving pruning of positive and negative log messages.
• Utilizing Gaussian mixture models (GMM) for reliable positive log selection.
• Iterative application of K-means, GMM, and Dirichlet process GMM for negative log selection.
• Employing a deep learning long short-term memory (LSTM) network for final anomaly detection.

Main Results:

• High precision achieved for both positive and negative log pruning.
• The proposed hybrid model demonstrates superior performance compared to established algorithms.
• Effective anomaly detection demonstrated on benchmark datasets (BGL, Openstack, Thunderbird).

Conclusions:

• The hybrid log anomaly detection technique effectively addresses the challenges of unstructured log data.
• Pruning reliable logs significantly enhances the precision of anomaly detection.
• The deep learning-based approach offers a robust solution for system log anomaly detection.