The perils of adding up: a cautionary tale about modeling and regression analysis

Area of Science:

• Health economics
• Policy analysis
• Statistical modeling

Background:

• Policy decisions often rely on statistical estimates derived from incomplete or imperfect data.
• The use of statistical relations for policy requires careful consideration beyond mere correlation.
• Ensuring the validity of estimates is crucial for reliable policy-making.

Purpose of the Study:

• To delineate the criteria for valid statistical modeling in policy contexts.
• To highlight the potential pitfalls of using non-theoretical or unconstrained statistical relations.
• To illustrate the dangers of ad hoc estimation methods with a real-world example.

Main Methods:

• Conceptual analysis of statistical modeling principles.
• Examination of theoretical requirements for valid models.
• Case study analysis of the World Health Report 2000's imputed "responsiveness" values.

Main Results:

• Statistical relations alone do not constitute a model; underlying theory is essential.
• Numerical estimates must respect definitional and accounting identities.
• Ad hoc methods can lead to uninterpretable or meaningless results, particularly when combining separately estimated partial values.

Conclusions:

• Valid policy-relevant modeling necessitates a theoretical foundation and respect for constraints.
• The World Health Report 2000's "responsiveness" estimates serve as a cautionary example of flawed modeling.
• Rigorous methodological standards are vital for generating trustworthy policy-related quantitative insights.