Best subset selection meets resampling: tackling selection instability at very low numbers of events per variable

Trust is a basic necessity for translating machine learning models from bench to bedside. To foster clinicians' trust in predictive models, these models should be interpretable or explainable. A major prerequisite for interpretability or explainability is that feature subsets are stable, i.e., that predictors returned by feature or best subset selection approaches do not change dramatically when the data is perturbed only slightly. However, prior research has shown that stable selections are usually not possible when the number of labels in the smaller class per feature, or outcome events per variable (EPV), is very low, and it remains challenging to build stable, interpretable models in such cases. In this work, we conduct a narrative review in the very low EPV regime and propose to combine cardinality-constrained best subset selection for logistic regression with resampling to tackle the selection instability. We perform a comprehensive evaluation on both synthetic and real-world clinical data, and find that our novel resampling-based model selection procedure yields interpretable feature subsets with a better discriminative performance and stability compared to best subset selection without resampling as well as feature selection with and without resampling. A Python package implementing the proposed methods is available at https://github.com/ml-lab-htw/l0l2learn.