A comparison of three strategies for scale construction to predict a specific behavioral outcome

Using 65 items from a mental health screening questionnaire, the History Opinion Inventory-Revised (HOI-R), the present study compared three strategies of scale construction-(1) internal (based on factor analysis), (2) external (based on empirical performance) and (3) intuitive (based on clinicians' opinion)-to predict whether 203,595 U.S. Air Force trainees would be discharged early for mental health or behavior-related reasons within a 4-year follow-up period. The external strategy significantly outperformed the internal strategy, which in turn outperformed the intuitive strategy. For all strategies, predictive accuracy was consistently higher when items and scales were scored using empirically derived weights rather than unit weights. These findings suggest that the external strategy of scale construction provides the highest accuracy when sample sizes are large and the aim is to predict a specific behavioral event. However, the internal strategy also yields valid results and can be a reasonable choice when outcome data are unavailable.