LAFS: A Fast, Differentiable Approach to Feature Selection Using Learnable Attention

Area of Science:

• Machine Learning
• Data Science
• Artificial Intelligence

Background:

• Feature selection is crucial for mitigating dimensionality but faces a speed-accuracy trade-off.
• Filter methods are fast but suboptimal; wrapper methods are powerful but slow.

Purpose of the Study:

• Introduce Learnable Attention for Feature Selection (LAFS), a novel framework for efficient and accurate feature selection.
• Achieve wrapper-level performance with the speed of simpler models.

Main Methods:

• LAFS utilizes a neural attention mechanism for context-aware feature importance scoring in a single pass.
• A hybrid loss function combines classification objective with an entropic regularizer for sparse, non-redundant feature selection.

Main Results:

• LAFS demonstrates strong performance on high-dimensional benchmark datasets, identifying complex feature interactions.
• The framework effectively handles multicollinearity and achieves results comparable to state-of-the-art methods like RFE-LGBM.

Conclusions:

• LAFS establishes a new accuracy-efficiency frontier in feature selection.
• Attention-based architectures offer a viable solution for the feature selection problem.