Improving reverse correlation analysis of faces: Diagnostics of order effects, runs, rater agreement, and image pairs

Examinations of the reliability and validity of classification images of faces using the reverse correlation approach remain rare. In the present paper, we focus on order effects of trials, compliance, and reliability effects, as well as the degree of contextual contrast of image pairs. We present different diagnostic methods to examine these three aspects using data from 12 reverse correlation studies conducted both in-lab and online with diverse samples (i.e., from Burkina Faso, China, the Netherlands, the U.S., and an international sample) using five different base faces (i.e., female black, female Asian, female and gender-neutral white, and black/white/female/male morphed composite). For each of the 12 studies, we compare the individual CIs of subgroups of likely non-complier respondents and trials with non-contrastful image pairs to individual CIs of likely compliers and contrastful image pairs. In an appendix, we also examine the effects of filtering out data from individual participants and trials on the signal-to-noise ratio of group CIs. R scripts are publicly available for easy implementation of our suggestions in related research.