Reliability of the Part II Board Certification Examination in Psychiatry: interexaminer consistency

The aim of the study was to examine the reliability (interexaminer consistency) of the American Board of Psychiatry and Neurology (ABPN) Part II (oral) examination in psychiatry.

Grades were assigned independently by two examiners who observed the same examination in a 1-year cycle (1,422 candidates, two examinations each). The consistency between these pairs of grades (pass, condition, fail) was analyzed using a weighted kappa statistic.

There was perfect agreement between examiners in 67% of examinations, minor disagreement in 26%, and major disagreement in 7% (weighted kappa = 0.54-0.56).

The Part II ABPN examination demonstrates fair to good reliability as measured by interexaminer consistency. Development of more explicit grading criteria should further improve examiner agreement in future examinations.