Designing a Streaming Algorithm for Outlier Detection in Data Mining-An Incrementa Approach.

Area of Science:

• Data Science and Machine Learning
• Algorithm Design and Analysis
• Real-time Data Processing

Background:

• Outlier detection in streaming data is crucial for applications like fraud detection and network monitoring.
• Existing algorithms struggle with streaming data's properties: concept drift, temporal context, transiency, uncertainty, and resource constraints.
• Incremental processing and scalability are key challenges for accurate outlier detection in real-time environments.

Purpose of the Study:

• To design and evaluate novel algorithms for accurate and efficient outlier detection in streaming data.
• To address the limitations of existing methods in handling the dynamic and resource-constrained nature of data streams.
• To introduce incremental approaches for established outlier detection techniques to enable real-time application.

Main Methods:

• Developed C_KDE_WR using sliding windows and kernel functions for online processing on GPUs (CUDA framework).
• Introduced C_LOF, an incremental adaptation of the Local Outlier Factor (LOF) algorithm, incorporating sliding windows and statistical summaries.
• Employed Precision, Recall, F-score, ROC curves, AUC, and t-tests for comparative accuracy and significance evaluation.

Main Results:

• C_KDE_WR demonstrated high throughput and effectiveness in handling real-time streaming data.
• C_LOF successfully adapted LOF for streaming contexts, overcoming high complexity and addressing the 'masquerading' problem.
• Both algorithms showed effectiveness and efficiency, with C_LOF offering significant improvements in accuracy and timely results.

Conclusions:

• C_KDE_WR and C_LOF are effective solutions for outlier detection in streaming data, addressing key challenges.
• The proposed incremental approaches enable real-time application of sophisticated outlier detection techniques.
• The developed algorithms offer a promising direction for robust and scalable outlier detection in dynamic data environments.