Enough?

Area of Science:

• Statistics
• Sociology
• Causal Inference

Background:

• Aronow et al. (2021) posited that randomized controlled trials (RCTs) are sufficient for causal inference, while nonparametric methods for observational studies are not.
• The concept of "enough" in statistical sufficiency is argued to be a sociological claim tied to institutional processes like peer review, rather than a purely statistical one.
• Existing statistical practice often relies on intuition regarding the complexity of real-world data, particularly concerning propensity score functions.

Purpose of the Study:

• To critically evaluate the claim that randomized controlled trials (RCTs) are "enough" for causal inference.
• To argue for a broader conception of statistical sufficiency that includes empirical knowledge of social context.
• To advocate for metascientific investigation into the complexity of causal inference in observational studies.

Main Methods:

• Critical analysis of the definition and implications of "enough" in statistical sufficiency.
• Examination of the sociological and institutional factors influencing statistical practice.
• Application of the critique to recommender systems as a case study, highlighting the limitations of intuition-based approaches.

Main Results:

• The sufficiency of statistical methods is not solely determined by statistical logic but by their integration with contextual knowledge.
• A complete understanding of "enough" requires knowledge beyond propensity scores, encompassing diverse spatial and temporal social characteristics.
• The practical utility of statistical critiques depends heavily on the specific context of application, such as recommender systems.

Conclusions:

• Statisticians should move beyond intuition and conduct empirical metascience to understand the complexity of causal inference in observational data.
• The reliance on "free lunches" in statistics, often limited to profitable applications in tech firms, should be questioned.
• Practicing statisticians must explicitly consider the "for what?" and "for whom?" questions to adequately address the sufficiency of their methods.