Real-World Data Difficulty Estimation with the Use of Entropy.

Area of Science:

• Data Science
• Information Theory
• Machine Learning

Background:

• The Internet of Things and big data generate overwhelming information, challenging traditional data analysis methods.
• Existing techniques struggle to efficiently extract non-redundant information, hindering effective decision-making.
• Selecting optimal solutions from complex datasets remains a significant challenge for practitioners.

Purpose of the Study:

• To investigate the utility of entropy measure as an indicator of data complexity and difficulty.
• To develop a method for capturing non-redundant, non-correlated core information from diverse real-world datasets.
• To enhance classification performance by leveraging entropy-based feature selection.

Main Methods:

• Data preprocessing applied to diverse real-world datasets (markets, sports, fake news).
• Entropy measure utilized to quantify data difficulty and identify core information.
• Classification algorithms applied to assess solution quality before and after entropy-based feature selection.
• Comparison of classification results using the best 25% of entropy-selected attributes versus initial preprocessing.

Main Results:

• Entropy measure effectively indicates data difficulty and identifies key information.
• Feature selection based on entropy significantly improves the quality of classification outcomes.
• The proposed entropy-based approach successfully extracts non-redundant information from complex datasets.

Conclusions:

• Entropy measure is a valuable tool for managing big data complexity and improving information extraction.
• Entropy-based feature selection offers a robust method for enhancing machine learning model performance.
• This approach provides a more efficient way to derive satisfactory solutions from large, complex datasets.