Using qualitative data for estimating construct validity of standardized measures

This paper describes efforts to develop a method of using qualitative interview data for investigating the construct validity of standardized measures. Illustration is provided through a multimethod validation study that examined the concurrence between observer assessments of interview data and a standardized paper and pencil measure-the Demands of Immigration Scale (DI). Specific aims of the multimethod validation study included (1) determining interrater reliability of the Demands of Immigration Observer Rating Scale (DI-ORS), which assessed respondents according to the interview data they provided and (2) establishing concurrence between observer assessments on the DI-ORS and the respondents' self-ratings on the DI scale. Interrater reliability of the DI-ORS was achieved with 97.9% agreement. However, there were significant differences (p < or = .05, 2-tailed) or disagreement on 5 out of 6 possible occurrences for agreement or disagreement between observer ratings on the DI-ORS and respondents' self ratings on the DI scale. The findings are informative for researchers who wish to use qualitative methods for construct validation.