Consensus analysis: reliability, validity, and informant accuracy in use of American and Mandarin Chinese pain...

A quantitative method for validating qualitative interview results and checking sample parameters is described and illustrated using common pain descriptions among a sample of Anglo-American and mandarin Chinese patients and dentists matched by age and gender. Assumptions were that subjects were members of a sociocultural group (e.g. ethnic or professional/lay) and answered questions independently about a monotonic domain (e.g. pain). Subjects answered 18 true/false items derived from and selected to reflect pain perceptions consistent with published and unpublished interview data. Estimates of consistency in use of descriptors within groups, validity of description, accuracy of individuals compared with others in their group, and minimum required sample size were calculated using Cronbach's alpha, factor analysis, and Bayesian probability. Ethnic and professional differences within and across groups were also tested using multidimensional scaling (MDS) and hypothesis testing. Consensus (consistency of subject response by group) was .99 among Anglo-Americans and .97 among Chinese. Mean subject accuracy was .81 for Americans and .57 for Chinese, indicating the need for larger numbers of Chinese to supplement each others' statements. However, larger numbers of subjects were recruited than actually required for each ethnic group at .95 confidence limits. MDS showed similarities in use of descriptors within ethnic groups, while there were differences (p < .001) between Chinese and American groups. Use of covalidating questionnaires that reflect results of qualitative interviews are recommended in order to estimate sample parameters such as intersubject agreement, individual subject accuracy, and minimum required sample sizes.