Reliability of short-listing in medical student selection

One in eight Universities Central Council on Admissions (UCCA) applications for admission to St Mary's Hospital Medical School in 1986 were in due course recirculated to the four short-listers, being seen again either by the same short-lister or by another short-lister. Intrarater reliabilities were high, not only for measures of educational achievement, but also for the more subjective assessments. Interrater reliabilities were more variable, being very high for educational achievement, but rather lower for the more subjective items, and being lowest for the assessment of 'Interests', suggesting divergence between short-listers' perceptions of the terms. Nevertheless, all reliabilities were sufficiently high to justify the continued use of these criteria during selection.