Fast sparse representative tree splitting via local density for large-scale clustering.

Area of Science:

• Data Mining
• Machine Learning
• Computational Statistics

Background:

• Large-scale clustering is crucial but challenging, with existing methods often failing to balance efficiency, accuracy, and adaptability.
• Parameter dependency and computational complexity hinder the application of current algorithms to massive datasets.

Purpose of the Study:

• To develop a novel, parameter-free framework for efficient and accurate large-scale clustering.
• To address the limitations of existing clustering algorithms in handling large, complex, and arbitrarily shaped data distributions.

Main Methods:

• A parameter-free cluster discovery approach using dynamic density-based splitting.
• A hybrid sampling-partitioning strategy integrating randomized sampling with K-means for linear complexity representative point extraction.
• Local density-driven Minimum Spanning Tree (MST) segmentation for adaptive partitioning based on density peaks.

Main Results:

• The proposed framework achieves higher clustering accuracy compared to state-of-the-art methods on large synthetic and real-world datasets (up to 20 million samples).
• Demonstrated significant reduction in runtime, showcasing superior efficiency.
• Exhibited robustness to varying sampling ratios and eliminated the need for user-specified parameters.

Conclusions:

• The novel framework offers a robust, efficient, and accurate solution for large-scale clustering challenges.
• Its parameter-free nature and adaptability make it highly suitable for real-world data mining applications with complex data structures.