Psychometric Properties of ABPN-Style Oral Examinations Administered Jointly by Two Psychiatry Residency Programs

The authors examined the reliability and validity of American Board of Psychiatry and Neurology-style oral exams (ASOs) given jointly by two schools. ASOs were given to 80 psychiatry residents from all four postgraduate years during the study period. Interrater reliability was near perfect for two years at one university, but it was much weaker at the second site. Test-retest reliability could not be demonstrated at either institution. Concurrent validity was demonstrated one year but was inconsistently demonstrated the second year. Likewise, predictive and construct validity were variable. International medical graduates were more apt to fail than American medical graduates. Although resident and faculty satisfaction with the examination was high, site-to-site variability in reliability and validity indicates the need to improve this ASO before using it for high-stakes evaluations.