Semantic response consistency and protocol validity in structured personality assessment: the case of the NEO-PI-R

In this study we tested the hypothesis that groups of NEO Personality Inventory-Revised (NEO-PI-R; Costa & McCrae, 1992a) protocols identified as potentially invalid by an inconsistency scale (INC; Schinka, Kinder, & Kremer, 1997) would show reduced reliability and validity according to a series of psychometric tests. Data were obtained from 2 undergraduate student samples, a self-report group (n = 132) who provided NEO-PI-R self-ratings on 2 occasions separated by a 7- to 14-day interval and an informant group (n = 109) who provided ratings of well-known friends or relatives on 2 occasions separated by a 6 month interval. INC scores from the Time 1 protocols were used to divide these samples into low, moderate, and elevated inconsistency groups. In both samples, these 3 groups showed equivalent levels of reliability and validity as measured by: contingency coefficients for the 20 INC item responses across occasions; test-retest intraclass correlations of NEO-PI-R domain scores; convergent correlations with Goldberg's (1992) Bipolar Adjective Scale scores; and discriminant correlations between the 5 NEO-PI-R domain scores. The similarity of results across self-report and informant assessment contexts provides additional evidence that semantic consistency approaches to assessing protocol validity may overestimate the prevalence of random or careless response behavior in standard administration conditions. Several theories are discussed that accommodate the existence of valid inconsistency in structured personality assessment.