A Hybrid Approach to Clustering in Big Data

Area of Science:

• Data Science
• Machine Learning
• Big Data Analytics

Background:

• Clustering big data is a significant challenge in data science.
• Existing algorithms like k-means and CURE have limitations, including initialization sensitivity and computational cost.

Purpose of the Study:

• Introduce and evaluate the novel clusiVAT algorithm for big data clustering.
• Compare clusiVAT's performance against established clustering methods.

Main Methods:

• clusiVAT employs data sampling, visual distance matrix analysis, and nearest prototype rule for non-iterative label extension.
• Comparison algorithms include k-means, single pass k-means, online k-means, and Clustering Using REpresentatives (CURE).

Main Results:

• clusiVAT demonstrates superior accuracy, recovering 97% of ground truth labels on the KDD-99 cup dataset.
• clusiVAT is significantly faster than other methods, processing 4.3 million samples in 76 seconds.
• k-means variants showed numerous failures due to initialization issues.

Conclusions:

• clusiVAT offers a robust, efficient, and accurate solution for big data clustering.
• The algorithm's independence from initialization makes it more reliable than k-means.
• clusiVAT surpasses CURE in speed while maintaining high accuracy.