Online and Unsupervised Anomaly Detection for Streaming Data Using an Array of Sliding Windows and PDDs

Area of Science:

• Data Science
• Machine Learning
• Signal Processing

Background:

• Anomaly detection in streaming data is crucial for identifying unusual patterns in real-time.
• Existing methods often require supervision or struggle with concept drift and contextual anomalies.
• The Numenta anomaly benchmark is a standard for evaluating anomaly detection algorithms.

Purpose of the Study:

• To propose an online and unsupervised anomaly detection algorithm for streaming data.
• To enhance the detection of contextual anomalies and concept drifts.
• To achieve superior performance compared to existing state-of-the-art algorithms.

Main Methods:

• Utilizing an array of sliding windows to segment streaming data into subwindows.
• Employing probability density-based descriptors (PDDs) with dimension reduction via kernel density estimation.
• Implementing a distance-based rule for anomaly detection.

Main Results:

• The proposed algorithm demonstrates competitive performance on the Numenta anomaly benchmark.
• Achieved higher detection rates for anomalies compared to the Hierarchical Temporal Memory (HTM) algorithm.
• Showcased earlier detection of contextual anomalies and concept drifts.

Conclusions:

• The developed algorithm provides an effective online and unsupervised approach for anomaly detection in streaming data.
• PDDs with dimension reduction offer a robust method for density estimation in subwindows.
• The algorithm shows significant potential for real-world applications requiring real-time anomaly identification.