The quantiles of extreme differences matrix for evaluating discriminant validity

When data on multiple indicators of underlying psychosocial constructs are collected, they are often intended as closely related assessments of a relatively unified phenomenon, or alternatively as capturing distinct facets of the phenomenon. Establishing distinctions among construct phenomena, assessments, or indicators is sometimes described as establishing discriminant validity. In the philosophical literature, often extreme instances or limit cases, actual or hypothetical, are used to identify settings in which one phenomenon is present and the other is not, to establish distinctions. We put forward an empirical analogue of this philosophical principle applied to distinctions amongst survey item responses. The quantiles of extreme differences matrix characterizes, for each pair of indicators, how large differences are between indicators at relatively extreme quantiles of the distribution of those differences. We discuss potential uses and properties of this matrix and related matrices for identifying relevant distinctions among indicators or facets of underlying construct phenomena.