The dysfunctional federally mandated proficiency test in cytopathology: a statistical analysis

Proficiency testing in cytopathology and in other disciplines should be based on firm statistical and scientific foundations, because test theory in general is a heavily statistical subject. Statistical considerations have demonstrated that the design of "short" proficiency tests in cytopathology, including the current federally mandated test, fundamentally is unsound because of the lack of sufficient validity and reliability. Examinees too frequently are misclassified by such short-format tests: Competent examinees fail the test in surprisingly high numbers, whereas most of the examinees who have insufficient cytologic skills eventually pass the test after the allowed retakes. Only dichotomous tests are suitable for accurate computation of the effects of test design on reliability, but the statistical conclusions also are generalizable to nondichotomous tests. In conclusion, the current federally mandated proficiency test cannot reliably measure the level of expertise of cytologists and, thus, cannot assure that only adequately skilled individuals evaluate Papanicolaou test samples. To render the test suitable for its intended purpose, the authors believe that complete redesign of the test, with the participation of experts in modern test theory, would be advisable.