Self-selection as an explanation for general mental ability test score differences between mobile and nonmobile...

Area of Science:

• Psychometrics
• Cognitive Psychology
• Human Resources

Background:

• Conflicting results exist on mobile device impact on general mental ability (GMA) test scores.
• Observational and lab studies show inconsistent GMA score differences between mobile and nonmobile device users.

Purpose of the Study:

• Investigate selection bias as a cause for inconsistent GMA score differences between mobile and nonmobile device users.
• Quantify the impact of selection bias on GMA test score disparities.

Main Methods:

• Analyzed operational data from 76,948 applicants across diverse occupations.
• Employed propensity score weighting and poststratification to control for selection bias.
• Compared GMA score differences before and after bias correction.

Main Results:

• Initially, mobile test-takers scored 0.58 SD lower than nonmobile users.
• Mobile device use correlated with lower educational attainment and job complexity.
• Propensity score weighting reduced the effect to d = 0.25; poststratification yielded d = 0.10.

Conclusions:

• Observed GMA score differences between devices in prior studies are likely inflated by selection bias.
• Propensity score weighting and poststratification effectively mitigate selection bias in real-world data.
• Random assignment is recommended for future GMA test evaluations to prevent selection bias.