Related Experiment Videos
Approximating Dunn's Cluster Validity Indices for Partitions of Big Data
IEEE Transactions on Cybernetics
|July 12, 2018
Summary
Dunn's index (DI) for large datasets is computationally expensive. This study introduces six approximation methods, including Maximin sampling and support vector machines, to efficiently estimate DI for better cluster analysis.
Area of Science:
- Data Mining
- Cluster Analysis
- Machine Learning
Background:
- Dunn's index (DI) is crucial for evaluating cluster partition quality.
- Calculating DI for large datasets (n objects) is computationally intensive due to its O(pn^2) complexity.
- Existing methods struggle with scalability for high-dimensional data.
Purpose of the Study:
- To develop computationally efficient methods for approximating Dunn's index (DI).
- To assess the accuracy and tractability of novel DI approximation techniques.
- To enable robust cluster validity assessment for large-scale datasets.
Main Methods:
- Four DI approximation methods based on Maximin sampling to identify cluster boundary points.
- Two DI approximation methods utilizing unsupervised one-class support vector machines.
- Comparative analysis of six approximation methods against exact DI computation.
Main Results:
- The proposed approximation methods, particularly those using incremental, neighborhood-based Maximin skeletons, demonstrate tractability.
- Numerical examples and experiments on seven datasets show reliable accuracy for the approximated DI.
- Maximin sampling and support vector machine approaches offer viable alternatives for large-scale cluster analysis.
Conclusions:
- Approximating Dunn's index is feasible and accurate for large datasets.
- Maximin sampling-based methods provide an efficient and reliable approach to cluster validity assessment.
- These methods enhance the practicality of DI for real-world, large-scale data mining applications.