A quantitative benchmark of neural network feature selection methods for detecting nonlinear signals.

Area of Science:

• Machine Learning
• Data Science
• Computational Statistics

Background:

• Feature selection (FS) is crucial for handling noisy datasets, especially with limited samples.
• Traditional FS methods often assume linear or additive feature relationships.
• Deep learning (DL) models offer potential for simultaneous FS and non-linear prediction.

Purpose of the Study:

• To systematically evaluate DL-based FS methods on synthetic datasets of varying complexity.
• To benchmark the efficacy of DL methods in uncovering non-linear feature relationships.
• To assess the reliability of gradient-based feature attribution techniques like Saliency Maps (SM) for Neural Networks (NNs).

Main Methods:

• Utilized synthetic datasets with controlled complexity to test FS and feature attribution methods.
• Compared performance of various DL-based FS techniques against traditional methods (Random Forests, TreeShap, mRMR, LassoNet).
• Benchmarked Saliency Maps (SM) for Neural Networks (NNs) using the same synthetic data settings.

Main Results:

• Most DL-based FS and SM methods were significantly challenged even by simple synthetic datasets.
• Random Forests, TreeShap, mRMR, and LassoNet demonstrated superior performance as FS methods.
• The reliability of DL-based FS and SM interpretation methods for non-linear features was found to be limited.

Conclusions:

• DL-based FS and SM interpretation methods are currently unreliable for identifying non-linear features in noisy, high-dimensional datasets.
• Traditional FS methods, particularly Random Forests and LassoNet, are more robust for such challenging scenarios.
• Further research is needed to improve the reliability of DL approaches for complex feature selection and interpretation tasks.