[Effect strength variation in the single group pre-post study design: a critical review]

Area of Science:

• Rehabilitation Research
• Psychometrics
• Statistical Analysis

Background:

• German rehabilitation research frequently employs uncontrolled one-group pre-post designs for evaluation and quality assessment.
• Outcome assessment typically involves comparing pre- and post-treatment means, with significance testing and effect size reporting on the rise.

Purpose of the Study:

• To present and critically evaluate various effect size indices applicable to the one-group pre-post design.
• To address fundamental concerns regarding the internal validity of this design in effect size calculations.

Main Methods:

• Analysis of different effect size index calculations for the one-group pre-post design.
• Focus on variations in the standardization denominator (pre-treatment SD, pooled SD, difference SD).
• Illustrative examples demonstrating divergent outcome patterns based on calculation methods.

Main Results:

• The numerator for all indices is the difference between pre- and post-treatment means.
• Significant variations in calculated effect sizes arise from different denominator choices.
• Effect sizes from one-group pre-post designs tend to be inflated compared to those from controlled studies.

Conclusions:

• The choice of standardization method critically impacts effect size interpretation in one-group pre-post designs.
• The limited internal validity of this design necessitates careful consideration of effect size reporting.
• Discussion of pros and cons of different indices, offering recommendations for best practices.