Assessing the inter-rater reliability and agreement of a hybrid assessment framework in the Emergency Department

There is often little consensus on the levels of trainees' performances, given the varied expertise and experience of the assessors, different tasks to be assessed and dynamic circumstances. Consequently, assessors are considered to be a major source of measurement error in workplace assessment. The aim of this pilot study is to assess the inter-rater reliability and agreement of a hybrid framework for assessment of nonspecialist doctors in our Emergency Department.

A hybrid framework was used to evaluate two core competencies of medical knowledge and patient care of nonspecialist doctors in our Emergency Department. Pre and post scores of these doctors were obtained at the end of their posting.

The inter-rater agreement and reliability was markedly higher for the group of nonspecialist doctors with previous Emergency Medicine experience, with fair-to-good reliability of the ratings among the assessors after application of the framework.

The use of our hybrid framework improved the inter-rater reliability and agreement in our Emergency Department for a specific group of learners who have had previous Emergency Medicine experience. To improve the reliability of ratings, faculty development has to address how observations and assessments should be synthesized into an overall rating.