Categorical variables with many categories are preferentially selected in bootstrap-based model selection procedures...

Area of Science:

• Statistics
• Biostatistics
• Computational Statistics

Background:

• Automated variable selection, like backward elimination, is common in multivariable regression.
• Bootstrap methods are used to assess the stability of these selection procedures.
• Categorical predictor variables pose unique challenges in variable selection.

Purpose of the Study:

• To investigate bias in automated variable selection with categorical predictors using bootstrap samples.
• To identify and recommend solutions for issues arising from categorical variables with varying numbers of categories.
• To compare the performance of bootstrap samples versus subsamples in variable selection.

Main Methods:

• Systematic assessment of automated variable selection using likelihood ratio tests.
• Comparison of bootstrap samples (with replacement) and subsamples (without replacement).
• Extensive simulations and analysis of a real-world dataset (NHANES).

Main Results:

• Automated variable selection on bootstrap samples significantly favors variables with more categories.
• This bias occurs even when variables have no true effect.
• Variables with many categories but no effect may be incorrectly preferred over variables with few categories but a real effect.

Conclusions:

• Bootstrap sampling introduces a bias favoring categorical variables with more levels in automated selection.
• This bias can lead to incorrect model selection, prioritizing irrelevant variables.
• Using subsamples instead of bootstrap samples is recommended to mitigate this selection bias.