Reliability of the Part II board certification examination in psychiatry: examination stability

Area of Science:

• Medical Education
• Psychiatry Board Certification
• Assessment Reliability

Background:

• The American Board of Psychiatry and Neurology (ABPN) Part II examination is crucial for psychiatric certification.
• Ensuring the reliability and stability of this high-stakes assessment is paramount for fair evaluation.
• Previous studies have explored various aspects of examination quality, but specific stability factors require ongoing investigation.

Purpose of the Study:

• To evaluate the examination stability of the ABPN Part II oral examination in psychiatry.
• To determine if factors such as examination format, site, day, or time influence grading consistency.

Main Methods:

• Analysis of examiner agreement using a weighted kappa statistic over a one-year cycle.
• Comparison of consistency across different examination components (live patient vs. videotape).
• Assessment of variations in agreement based on examination site, day, and time of administration.

Main Results:

• No significant differences in examiner agreement were found across different examination parts.
• Consistency of grading was not significantly affected by the examination site, day of the week, or time of day.
• The analysis indicated that candidate performance is the primary driver of pass/fail decisions.

Conclusions:

• The ABPN Part II oral examination in psychiatry exhibits robust stability.
• Examination format and site of administration do not significantly impact the reliability of the assessment.
• Candidate performance remains the most influential factor in determining examination success or failure.