An Entropy-Based Clustering Algorithm for Real-Time High-Dimensional IoT Data Streams.

Area of Science:

• Data Science
• Machine Learning
• Signal Processing

Background:

• The proliferation of sensors and Internet of Things (IoT) devices generates massive high-dimensional data streams.
• Real-time clustering of this data faces challenges due to dimensionality, memory, and time constraints.
• Existing dimensionality reduction techniques often lack effective feature ranking, impacting clustering performance.

Purpose of the Study:

• To introduce E-Stream, a novel entropy-based clustering algorithm for efficient real-time processing of high-dimensional data streams.
• To address limitations of traditional clustering and dimensionality reduction methods in dynamic environments.
• To improve clustering accuracy and computational efficiency for IoT data.

Main Methods:

• E-Stream employs real-time feature ranking based on entropy within a sliding time window.
• The identified informative features are then used with the DenStream algorithm for clustering.
• Evaluation was conducted using the NSL-KDD dataset, comparing E-Stream against DenStream, CluStream, and MR-Stream.

Main Results:

• E-Stream demonstrated superior performance over baseline algorithms in clustering accuracy and computational efficiency.
• The algorithm effectively reduced dimensionality while requiring less memory and computational resources.
• Evaluation metrics included average F-Measure, Jaccard Index, Fowlkes-Mallows Index, Purity, and Rand Index.

Conclusions:

• E-Stream is a suitable algorithm for real-time processing of high-dimensional data streams, particularly in IoT applications.
• The entropy-based feature ranking significantly enhances clustering performance and efficiency.
• Future research will focus on developing a parameter-free version and enhancing adaptability to diverse and dynamic data.