Comparative clinical trials in psychotherapy: Have large effects been replicated?

Area of Science:

• Psychiatry and Mental Health
• Clinical Psychology
• Evidence-Based Practice

Background:

• Replication is crucial for scientific validity, especially in psychotherapy research.
• Large effect sizes in clinical trials can heavily influence meta-analysis outcomes.
• Previous psychotherapy trials with substantial effect sizes may not be reliably reproduced.

Purpose of the Study:

• To evaluate the extent of replication attempts for psychotherapy clinical trials demonstrating large effect sizes.
• To assess the impact of unreplicated large-effect trials on meta-analytic conclusions for depression and anxiety.
• To identify the need for standardized replication criteria in psychotherapy research.

Main Methods:

• Developed standard criteria to differentiate direct and conceptual replication methodologies.
• Conducted a literature search of meta-analyses comparing psychotherapies.
• Identified trials with large effect sizes (d > 0.8) from meta-analyses.
• Searched for cited direct and conceptual replications of these large-effect trials.

Main Results:

• Ten meta-analyses met inclusion criteria, identifying 12 distinct trials with large effect sizes.
• No direct replication attempts were found for trials with large effects.
• Four conceptual replications were identified, with only two partially corroborating original findings.
• Meta-analyses with multiple large-effect trials suggested treatment superiority.

Conclusions:

• The lack of direct replication for large-effect psychotherapy trials casts doubt on treatment superiority claims in meta-analyses.
• Conclusions from meta-analyses heavily influenced by unreplicated large effects are questionable.
• Establishing authoritative replication criteria and quality benchmarks is essential for improving clinical science and meta-analytic rigor.