Permutation importance: a corrected feature importance measure.

Area of Science:

• Life Sciences
• Bioinformatics
• Machine Learning

Background:

• Interpretability is crucial in life sciences, alongside prediction accuracy.
• Linear models are common for feature relevance but lack flexibility.
• Complex models like Support Vector Machines and Random Forests (RF) have advanced feature relevance estimators, but RF models exhibit bias towards categorical variables with many categories.

Purpose of the Study:

• To introduce a heuristic for normalizing feature importance measures to correct bias.
• To improve the interpretability and prediction accuracy of machine learning models, particularly RF.

Main Methods:

• Developed a heuristic for normalizing feature importance.
• Employed repeated permutations of the outcome vector to estimate importance distribution in a non-informative setting.
• Calculated P-values for observed importance to provide a corrected measure.

Main Results:

• Non-informative predictors did not receive significant P-values in simulated data.
• Informative variables were successfully recovered among non-informative ones.
• P-values from permutation importance (PIMP) significantly improved variable selection and model interpretability.
• An improved RF model using PIMP-selected variables showed superior prediction accuracy in real-world case studies.

Conclusions:

• The PIMP method effectively corrects feature importance bias in machine learning models.
• This approach enhances model interpretability by providing reliable variable significance.
• The proposed RF model incorporating PIMP demonstrates improved predictive performance.