Oblique and rotation double random forest.

Area of Science:

• Machine Learning
• Ensemble Methods
• Data Mining

Background:

• Random Forest (RF) is a powerful ensemble method utilizing bagging and random subspace concepts.
• The strength of RF lies in the diversity and instability of its base learners (decision trees).
• Standard RF uses univariate decision trees, which may not capture data's geometric structure effectively.

Purpose of the Study:

• To propose two novel ensemble methods: rotation double random forest and oblique double random forest.
• To enhance the diversity of base learners and improve generalization performance.
• To address limitations of standard RF, such as axis-parallel splits and suboptimal tree depth.

Main Methods:

• Rotation Double Random Forest: Applies feature space transformations (PCA, LDA) at each node for increased base learner diversity.
• Oblique Double Random Forest: Employs multivariate decision trees with optimal plane generation using multisurface proximal SVM and regularization techniques for small sample sizes.
• Bagging is applied at each non-leaf node to promote larger tree sizes and improved performance.

Main Results:

• Both proposed methods demonstrated improved performance compared to baseline models.
• Evaluation on 121 UCI benchmark datasets and real-world fisheries datasets confirmed the efficacy of the new models.
• Statistical analysis and experimental results validated the superiority of oblique and rotation double random forests.

Conclusions:

• The proposed rotation and oblique double random forest models offer significant improvements over standard ensemble methods.
• These advanced techniques effectively capture data's geometric properties and mitigate small sample size challenges.
• The enhanced diversity and structure of the proposed models lead to superior generalization performance.