Random forests with parametric entropy-based information gains for classification and regression problems.

Area of Science:

• Machine Learning
• Data Science
• Complex Systems

Background:

• Random forests are popular for classification and regression, excelling on tabular data.
• Traditional random forests use Shannon entropy, potentially limiting prediction accuracy.
• Deformed entropies from complex systems offer a novel approach to enhance these algorithms.

Purpose of the Study:

• To explore the efficacy of deformed entropies in improving random forest prediction accuracy.
• To introduce novel information gains based on Renyi, Tsallis, and Sharma-Mittal entropies for random forests.
• To evaluate these modifications on diverse classification and regression benchmark datasets.

Main Methods:

• Developed information gains using Renyi, Tsallis, and Sharma-Mittal entropies.
• Integrated these new information gains into random forest algorithms.
• Tested modified algorithms on six benchmark datasets (three classification, three regression).

Main Results:

• Classification: Renyi entropy improved accuracy by 19-96%, Tsallis by 20-98%, and Sharma-Mittal by 22-111%.
• Regression: Deformed entropies improved R2 prediction by 2-23% depending on the dataset.
• All tested deformed entropies outperformed the classical Shannon entropy-based random forest.

Conclusions:

• Deformed entropies offer a significant improvement over Shannon entropy for random forest accuracy.
• These novel approaches provide enhanced predictive power for both classification and regression tasks.
• The findings suggest a promising direction for advancing machine learning algorithms using concepts from complex systems.