Related Experiment Videos
Anomaly Detection for the Centralised Elasticsearch Service at CERN
Jennifer R Andersson1, Jose Alonso Moya2, Ulrich Schwickerath3
1Department of Information Technology, Uppsala University, Uppsala, Sweden.
Frontiers in Big Data
|December 6, 2021
Summary
CERN developed an anomaly detection system for its Elasticsearch service, using LSTM neural networks to process monitoring data. This system identifies potential issues in real-time, improving service reliability for users.
Area of Science:
- Distributed systems
- Data analytics
- Machine learning
Background:
- CERN provides a centralized Elasticsearch service for user data search and analytics.
- The service comprises ~30 independent Elasticsearch clusters supporting hundreds of users, primarily for logs and service analytics.
- The complexity generates vast internal monitoring data, challenging real-time processing with limited personnel.
Purpose of the Study:
- To develop an automated system for processing Elasticsearch monitoring data.
- To detect anomalies and potential issues in real-time.
- To enable proactive issue resolution for the Elasticsearch service.
Main Methods:
- Investigated both deep learning and traditional methods for data analysis.
- Deployed a one-layer, multi-dimensional LSTM neural network for anomaly detection.
- Utilized a simple moving average for validating anomaly detection results.
Main Results:
- Successfully implemented an anomaly detection system processing internal monitoring data.
- The system aims to identify and flag potential issues before they impact users.
- Experiences with actual data application are reported.
Conclusions:
- The developed system leverages LSTM neural networks for real-time anomaly detection in a large-scale Elasticsearch service.
- The approach facilitates proactive maintenance and enhances service stability.
- Future improvements are being considered to address system weaknesses.