CTD: Fast, accurate, and interpretable method for static and dynamic tensor decompositions.

Area of Science:

• Data Science
• Machine Learning
• Cybersecurity

Background:

• Detecting patterns and anomalies in multi-dimensional data (tensors) is crucial for applications like cybersecurity and network monitoring.
• Existing tensor decomposition methods lack interpretability, efficiency, and speed for real-time, large-scale data analysis.

Purpose of the Study:

• To propose a novel, fast, accurate, and directly interpretable tensor decomposition method for online pattern and anomaly detection.
• To introduce both static (CTD-S) and dynamic (CTD-D) versions of the proposed method to handle diverse data streams.

Main Methods:

• Developed CTD ( a tensor decomposition technique) utilizing efficient sampling for enhanced interpretability and performance.
• Introduced CTD-S (static version) with provable accuracy improvements and significant gains in speed and memory efficiency.
• Introduced CTD-D (dynamic version), the first interpretable dynamic tensor decomposition method, achieving substantial speedups through temporal factor exploitation.

Main Results:

• CTD-S demonstrates up to 11x higher accuracy, 2.3x faster processing, and 24x more memory efficiency than state-of-the-art methods.
• CTD-D achieves up to 82x speedup compared to CTD-S, enabling real-time analysis.
• Successfully applied CTD for online distributed denial of service (DDoS) attack detection and online troll detection.

Conclusions:

• CTD offers a significant advancement in interpretable tensor decomposition for real-time anomaly detection.
• The proposed method addresses key limitations of existing techniques, providing a practical solution for large-scale, dynamic data.
• CTD's effectiveness is validated through successful applications in critical cybersecurity scenarios.