Methods in causal inference. Part 3: measurement error and external validity threats.

Area of Science:

• Human sciences
• Comparative cultural research
• Causal inference

Background:

• Broad sampling beyond Western, educated, industrialized, rich, and democratic (WEIRD) societies is ethically and scientifically desirable for generalizability in human sciences.
• Restricting target populations is sometimes necessary but can introduce selection restriction bias, threatening valid causal inference.
• Existing statistical tests like invariance tests do not address structural biases in comparative cultural research.

Purpose of the Study:

• To clarify conditions under which unrestricted sampling is desirable or undesirable in human sciences.
• To use causal diagrams to identify structural features of measurement error and target population restriction bias.
• To define 'weird' studies as those with biases threatening causal inference in comparative cultural research.

Main Methods:

• Utilized causal diagrams to analyze measurement error bias and target population restriction bias.
• Focused on threats to valid causal inference in comparative cultural research.
• Examined the limitations of statistical invariance tests in addressing structural biases.

Main Results:

• Causal diagrams effectively clarify structural biases arising from measurement error and selection restriction.
• Defined 'weird' studies as those with wrongly estimated inferences due to inappropriate restriction and distortion.
• Demonstrated that statistical invariance tests cannot resolve these structural biases.

Conclusions:

• Comparative cultural research requires careful consideration of sampling strategies to avoid bias.
• Causal inference workflows offer essential checklists for conducting safe and effective cross-cultural research.
• Understanding and mitigating 'weird' biases are crucial for advancing generalizable knowledge in human sciences.