An automatic finite-sample robustness metric: when can dropping a little data change conclusions? Part I: definitions...

Area of Science:

• Statistics
• Econometrics
• Data Science

Background:

• Study samples often deviate from target populations due to sampling challenges or population shifts.
• Researchers must ensure that study conclusions are not overly sensitive to minor variations in sample data.

Purpose of the Study:

• To develop and present a computational method for assessing the sensitivity of applied conclusions to small sample perturbations.
• To quantify the impact of removing a small fraction of sample data on study inferences.

Main Methods:

• Proposed the 'Approximate Maximum Influence Perturbation' (AMIP) metric, derived from the classical influence function.
• Utilized an approximation to identify the most influential subset of data, overcoming computational infeasibility of exhaustive checks.
• Demonstrated AMIP's applicability to various statistical methods like OLS, IV, ML, GMM, and Variational Bayes.

Main Results:

• The AMIP metric provides a computable assessment of sensitivity for common statistical models.
• An exact finite-sample lower bound on sensitivity is provided with minimal additional computational cost.
• Empirical analysis showed that results from several influential economics papers were sensitive and could be overturned by removing less than 1% of the sample.

Conclusions:

• The study introduces a novel metric for evaluating the robustness of research findings to sample data variations.
• Findings highlight potential vulnerabilities in influential economic studies, underscoring the importance of sensitivity analysis.
• The proposed method offers a practical tool for researchers to enhance the reliability and validity of their statistical inferences.