A Novel Spark-Based Attribute Reduction and Neighborhood Classification for Rough Evidence

Area of Science:

• Data Science
• Machine Learning
• Artificial Intelligence

Background:

• Traditional Neighborhood Classification (NEC) algorithms often use majority voting, neglecting spatial differences and label uncertainty, leading to misclassification.
• Existing NEC methods struggle with large datasets due to inefficient in-memory processing.

Purpose of the Study:

• To develop a novel, efficient Spark-based NEC algorithm incorporating attribute reduction and rough evidence.
• To enhance classification accuracy and computational performance for large-scale datasets.

Main Methods:

• Constructed a multigranular sample space using parallel undersampling.
• Evaluated attribute significance using neighborhood rough evidence decision error rate for attribute reduction.
• Designed parallel attribute reduction and classification decision processes within the Spark framework.

Main Results:

• The proposed Spark-based NEC algorithm demonstrated superior classification accuracy compared to traditional methods.
• Significant improvements in computational efficiency were observed, particularly on large-scale datasets.
• The method effectively handles spatial differences and label uncertainty in neighborhood samples.

Conclusions:

• The novel Spark-based NEC approach offers a more accurate and computationally efficient solution for classification problems.
• Integrating attribute reduction and rough evidence effectively addresses limitations of traditional NEC algorithms.
• The parallelized framework is well-suited for processing large datasets in machine learning applications.