Inter-Rater Reliability of the Mentor Behavioral Interaction Rubric

Area of Science:

• Medical Education
• Psychology
• Social Sciences

Background:

• Objective assessment of mentor behavioral skills is crucial for evaluating mentor training interventions in academic settings.
• The Mentor Behavioral Interaction (MBI) Rubric is a novel, content-valid observational measure for mentor behavioral skills during single mentor-mentee interactions.

Purpose of the Study:

• To assess the inter-rater reliability (IRR) of the MBI Rubric when applied to video-recorded mentor-mentee interactions.

Main Methods:

• Three expert raters synchronously assessed 26 video-recorded mentor-mentee interactions using the MBI Rubric.
• The MBI Rubric comprises six scaled items (Part 1) and ten yes/no items (Part 2).
• Raters resolved disagreements through consensus or majority vote after initial individual scoring.

Main Results:

• Inter-rater reliability ranged from good (Part 1 IRR=0.67) to excellent (Part 2 IRR=0.83).
• Rater effects were observed in initial scoring (Part 1), but not in Part 2.
• Raters' final scores were higher than their initial individual scores for both parts.

Conclusions:

• The MBI Rubric demonstrates content validity and inter-rater reliability for assessing mentor behavioral skills in single interactions.
• It can be utilized alongside other instruments to measure mentor training efficacy.
• Limitations include potential ceiling effects and resource-intensive administration; future research will explore responsiveness and construct validity.