WEClustering: word embeddings based text clustering technique for large datasets.

Area of Science:

• Computer Science
• Data Mining
• Natural Language Processing

Background:

• Vast amounts of digital text data necessitate efficient data mining techniques.
• Traditional clustering methods struggle with the high dimensionality and sparsity of large textual datasets.
• Existing techniques like K-means, Agglomerative clustering, and DBSCAN are inadequate for large-scale text analysis.

Purpose of the Study:

• To propose a novel text clustering technique, WEClustering, specifically designed for large textual datasets.
• To address the limitations of traditional clustering algorithms in handling high dimensionality.
• To improve the accuracy and effectiveness of text categorization and topic extraction.

Main Methods:

• Utilizing word embeddings derived from the Bidirectional Encoders Representations using Transformers (BERT) deep learning model.
• Developing the WEClustering algorithm to effectively manage high-dimensional text data.
• Validating the proposed technique on diverse datasets of varying sizes.

Main Results:

• WEClustering effectively overcomes the challenges of high dimensionality in text data.
• The technique demonstrates superior performance compared to widely used and state-of-the-art clustering methods.
• Significant improvements were observed in cluster accuracy, as measured by Purity and Adjusted Rand Index.

Conclusions:

• WEClustering offers a robust and accurate solution for clustering large text datasets.
• The integration of BERT word embeddings enhances the capability of text data mining.
• This approach provides a significant advancement in topic extraction and information retrieval from massive text corpora.