Distributed K-Means algorithm based on a Spark optimization sample.

Area of Science:

• Data Science
• Machine Learning
• Big Data Analytics

Background:

• Classical K-Means algorithm suffers from instability and performance issues with massive datasets.
• Efficient clustering of large-scale data is crucial for various data mining applications.

Purpose of the Study:

• To introduce SOSK-Means, an enhanced K-Means algorithm optimized for Spark to address the limitations of classical K-Means on massive datasets.
• To improve the computational speed and accuracy of K-Means clustering for large-scale data.

Main Methods:

• Implemented a weighted jump-bank approach for efficient random sampling and pre-clustering, improving initial center selection.
• Utilized a weighted max-min distance with variance for enhanced distance calculation, considering data weight and variance.
• Employed a novel distance comparison method and a Directed Acyclic Graph (DAG) for optimized computation and distributed processing on Spark.

Main Results:

• SOSK-Means demonstrates significant improvements in computational speed compared to classical K-Means.
• The algorithm maintains high computational accuracy, effectively handling massive datasets.
• Enhanced initial center selection and distance calculation contribute to improved clustering performance.

Conclusions:

• SOSK-Means offers a robust and efficient solution for large-scale data clustering using Spark optimization.
• The proposed modifications effectively address the instability and performance bottlenecks of traditional K-Means.
• This optimized algorithm is well-suited for big data analytics requiring fast and accurate clustering.