Detecting Group Anomalies in Tera-Scale Multi-Aspect Data via Dense-Subtensor Mining.

Area of Science:

• Data Mining
• Machine Learning
• Big Data Analytics

Background:

• Dense subtensors in large-scale tensors often indicate fraudulent activities like botnets and network attacks.
• Existing methods for dense subtensor detection struggle with large datasets exceeding memory capacity, leading to low accuracy or infeasibility.
• Real-world applications in social media, web data, and network traffic generate massive tensors requiring efficient analysis.

Purpose of the Study:

• To develop a memory-efficient and scalable method for detecting fraudulent lockstep behavior in large-scale tensor data.
• To address the limitations of existing methods that assume data fits within main memory.
• To enable accurate detection of dense subtensors in datasets too large for disk storage.

Main Methods:

• Proposing D-Cube, a disk-based dense-subtensor detection algorithm.
• Implementing D-Cube for distributed processing across multiple machines.
• Ensuring provable accuracy guarantees for detected subtensor densities.

Main Results:

• D-Cube requires significantly less memory (up to 1,561x) and handles substantially larger data (1,000x, up to 2.6TB).
• The method achieves up to 7x speed improvement due to near-linear scalability.
• D-Cube accurately identified network attacks in TCP dumps and synchronized behavior in rating data, outperforming state-of-the-art methods.

Conclusions:

• D-Cube provides an effective, memory-efficient, and scalable solution for detecting dense subtensors in massive datasets.
• The disk-based and distributed nature of D-Cube makes it suitable for real-world large-scale data analysis.
• This approach enhances the detection of anomalous and fraudulent behaviors in various domains.