Whose criterion standard is it anyway?

A criterion-referenced standard is an important element of most successful professional testing programs. A growing body of evidence suggests that judge decisions are influenced by characteristics related to the normative experience of the individual judge (e.g. gender, age, etc.). This investigation used two health-care related boards to explore the effects of judge characteristics on the standards established. Two judge panels (composed of 26 and 30 members respectively) were used in a simplified Objective Standard Setting exercise to define examination cutoff points. Multi-faceted Rasch analyses were employed to detect and explore differences in judgment making. Significant but not necessarily consistent differences were found between panel judges on several examined characteristics. Results suggest that criterion-referenced standards defined by judge panels are inexorably connected to their normative experiences and are therefore wholly sample dependent. While stratification of judge panels is clearly an important element in defining standards, if they are ever to achieve the goals of Glaser (1963) and Majer (1962) including meaningful independence, more must be done to investigate these and other concerns. The case for the predictive validity of criterion-referenced standards has not thusfar been made in any convincing fashion.