Residual permutation tests for feature importance in machine learning.

Area of Science:

• Psychology
• Computer Science
• Statistics

Background:

• Traditional psychological research heavily utilizes linear models for hypothesis testing.
• Machine learning (ML) offers advanced methods for exploring complex, non-linear variable relationships.
• Current feature importance tools in ML lack robust statistical inference capabilities.

Purpose of the Study:

• To develop statistically sound methods for hypothesis testing within machine learning frameworks.
• To introduce residual permutation tests (RPTs) as a tool for assessing feature significance in ML models.
• To address the gap in inferential statistics for interpreting 'black-box' ML algorithms.

Main Methods:

• Introduced two variants of residual permutation tests: RPT on Y (RPT-Y) and RPT on X (RPT-X).
• RPT-Y permutes label residuals conditioned on other features.
• RPT-X permutes target feature residuals conditioned on other features.
• Conducted a comprehensive simulation study across diverse ML algorithms.

Main Results:

• RPT-X demonstrated stable empirical Type I error rates below the nominal level.
• RPT-X showed appropriate statistical power in both regression and classification tasks.
• The study validated RPT-X performance across a wide range of ML algorithms.

Conclusions:

• Residual permutation tests, particularly RPT-X, provide a valid approach for statistical inference in ML.
• RPT-X is a valuable tool for hypothesis testing, enhancing the interpretability of ML models.
• The findings support the broader adoption of RPT-X in psychological research and other ML applications.