Reliability and validity of gain scores considered graphically

Area of Science:

• Psychometrics
• Educational Measurement
• Statistical Analysis

Background:

• Gain scores (posttest minus pretest) are commonly used to measure change.
• Previous research has explored the reliability and validity of gain scores with varying results.
• Understanding the conditions influencing gain score properties is crucial for accurate interpretation.

Purpose of the Study:

• To systematically examine the impact of various psychometric parameters on the reliability and validity of gain scores.
• To extend and refine previous research on gain score evaluation.
• To provide clear guidelines on the appropriate use of gain scores in research.

Main Methods:

• Systematic variation of pretest/posttest reliabilities, validities, standard deviation ratios, and correlations.
• Graphical analysis of results to visualize complex relationships between parameters and gain score properties.
• Simulation-based approach to explore a wide range of parameter configurations.

Main Results:

• Gain score reliability can be maximized when validity is minimized, and vice versa.
• High pretest-posttest correlation enhances gain score validity but reduces reliability.
• Graphical representations offer clearer insights into parameter effects than tabular data alone.

Conclusions:

• Researchers must carefully consider psychometric properties before using gain scores.
• Specific conditions exist where gain scores are both reliable and valid, while other conditions render them unsuitable.
• Findings inform decisions on when to employ or avoid gain scores for accurate measurement of change.