LogEvent2vec: LogEvent-to-Vector Based Anomaly Detection for Large-Scale Logs in Internet of Things.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Log anomaly detection is crucial for managing large-scale Internet of Things (IoT) systems.
• Natural Language Processing (NLP) methods, like word2vec, are increasingly used for log feature extraction.
• Traditional word2vec has high computational costs and requires multiple transformations for log sequence analysis.

Purpose of the Study:

• To propose LogEvent2vec, an offline feature extraction model to reduce computational cost and avoid multiple transformations in log anomaly detection.
• To directly vectorize log events using word2vec principles, enabling efficient feature extraction.
• To integrate LogEvent2vec with various transformation methods and anomaly detection models.

Main Methods:

• LogEvent2vec takes log events as input for direct feature extraction and vectorization.
• Log event vectors are transformed into log sequence vectors using barycentric coordinate transformation or TF-IDF.
• Supervised models including Random Forests, Naive Bayes, and Neural Networks are trained for anomaly detection.

Main Results:

• LogEvent2vec significantly reduces computational time by 30 times compared to word2vec.
• The proposed model improves detection accuracy.
• LogEvent2vec combined with barycentric coordinate transformation and Random Forest achieved the best F1-score.
• LogEvent2vec with TF-IDF and Naive Bayes demonstrated the least computational time.

Conclusions:

• LogEvent2vec offers a computationally efficient and accurate approach to log anomaly detection.
• The model effectively extracts features from log events for sequence-based anomaly detection.
• LogEvent2vec provides flexibility in integration with different downstream anomaly detection techniques.