Related Experiment Videos
The homogeneity assumption in differential prediction analysis: does it really matter?
F L Oswald1, S Saad, P R Sackett
1Department of Psychology, University of Minnesota, USA. foswald@msu.edu
The Journal of Applied Psychology
|August 19, 2000
Summary
The F test for regression slopes may distort error rates if subgroup variances differ greatly. While many psychology datasets have similar variances, some heterogeneity exists, suggesting alternative statistical tests for differential prediction research.
Area of Science:
- Psychometrics
- Applied Psychology
- Statistical Methods
Background:
- The F test for differences in regression slopes is sensitive to unequal subgroup error variances, potentially distorting Type I and II error rates.
- A variance ratio exceeding 1.50:1 is a critical threshold where distortions become problematic.
- Previous simulation studies highlight the risks, but empirical data on the frequency of this issue in applied psychology is limited.
Purpose of the Study:
- To investigate the frequency and extent of unequal subgroup error variances in real-world datasets relevant to applied psychology.
- To determine if common datasets violate the 1.50:1 error variance ratio, impacting the validity of F test results.
- To provide empirical evidence to guide psychologists on the appropriate use of the F test in differential prediction studies.
Main Methods:
- Analysis of two large databases: the General Aptitude Test Battery (GATB) validity study and the Project A military database.
- Examination of subgroup error variances (e.g., White-Black, male-female) for ability and performance data.
- Calculation of the ratio of subgroup error variances to assess homogeneity.
Main Results:
- Subgroup error variances in both databases were often homogeneous enough to support previous empirical findings based on the F test.
- However, sufficient heterogeneity was observed in a notable number of cases to warrant caution.
- The 1.50:1 ratio was violated in some instances, particularly when considering specific subgroups and performance measures.
Conclusions:
- While the F test may be robust in many applied psychology contexts, heterogeneity of error variances is a present concern.
- Applied psychologists conducting differential prediction research should routinely examine their data for unequal subgroup variances.
- Consideration of alternative statistical tests is recommended when significant heterogeneity is detected to ensure accurate error rates.