The limitations of randomized controlled trials in predicting effectiveness

Area of Science:

• Health Services Research
• Social Policy Analysis
• Evidence-Based Practice

Background:

• Increasing reliance on studies employing Mill's method of difference (e.g., randomized controlled trials) for policy decisions.
• The concept of 'external validity' is commonly used to assess the generalizability of causal conclusions from such studies.
• Current evidence grading guides often prioritize studies with strong methodological designs based on the method of difference.

Purpose of the Study:

• To critically evaluate the concept of 'external validity' in the context of health and social care interventions.
• To propose 'capacities' as a more appropriate framework for supporting effectiveness predictions.
• To highlight the implications of this shift for policy decision-making and evidence appraisal.

Main Methods:

• Conceptual analysis of evidence appraisal frameworks.
• Critique of the 'external validity' concept.
• Introduction of the 'capacities' framework for evaluating intervention effectiveness.
• Case study illustration using multisystemic therapy for youth antisocial behavior.

Main Results:

• External validity is argued to be an inadequate concept for assessing the generalizability of causal findings.
• 'Capacities' are proposed as a more suitable theoretical construct for supporting effectiveness predictions.
• Relying solely on method-of-difference studies for policy recommendations is insufficient when considering capacities.

Conclusions:

• The traditional focus on external validity hinders robust policy recommendations.
• A 'capacities'-based approach requires diverse and different types of evidence beyond method-of-difference studies.
• Rethinking evidence appraisal is crucial for effective health and social care policy, as demonstrated by the multisystemic therapy example.