Evaluation of Prediction-Oriented Model Selection Metrics for Extended Redundancy Analysis.

Area of Science:

• Statistics
• Machine Learning
• Data Analysis

Background:

• Extended redundancy analysis (ERA) is a statistical method linking multiple predictor sets to response variables.
• Conventional ERA model evaluation overestimates performance by using development data for assessment.
• Overfitting is a common issue in model selection due to optimistic performance estimates.

Purpose of the Study:

• Introduce a novel model evaluation approach for ERA using resampling methods.
• Develop new ERA metrics to assess out-of-sample predictive performance.
• Address the lack of research on out-of-sample evaluation in ERA.

Main Methods:

• Utilized computer-intensive resampling techniques for model evaluation.
• Developed and proposed new ERA model evaluation metrics.
• Compared the proposed approach with conventional methods using simulated and real data.

Main Results:

• The conventional ERA evaluation method favors complex models, leading to overfitting.
• The proposed resampling-based approach effectively identifies the true ERA model.
• Mis-specified models (underfitted and overfitted) were distinguished by the new method.

Conclusions:

• The proposed out-of-sample evaluation approach for ERA is superior to conventional methods.
• Resampling techniques provide a reliable way to prevent overfitting in ERA model selection.
• This research offers a robust framework for accurate ERA model assessment.