The Reliability of Rating via Audio-Recording Using the Mindfulness-Based Interventions: Teaching Assessment...

Area of Science:

• Mindfulness-based interventions research
• Clinical psychology
• Psychotherapy assessment

Background:

• The Mindfulness-Based Interventions: Teaching Assessment Criteria (MBI:TAC) assesses mindfulness teacher skills and intervention fidelity.
• Current MBI:TAC assessment relies on video recordings, posing challenges in acquisition, sharing, and participant privacy.
• Audio-only recordings present a potential alternative, but their reliability requires investigation.

Purpose of the Study:

• To evaluate evaluator perceptions of the rating process using audio-only recordings.
• To determine the inter-rater reliability of MBI:TAC assessments conducted with audio-only recordings.

Main Methods:

• Audio-only recordings were created from 21 video-recorded Mindfulness-Based Stress Reduction teachers.
• Twelve trained MBI:TAC assessors rated the audio recordings, with each teacher rated by three unique assessors.
• Semi-structured interviews were conducted with the evaluators to gather qualitative feedback.

Main Results:

• Intraclass correlation coefficients (ICCs) for audio ratings across 6 MBI:TAC domains ranged from .53 to .69 when using an average of three evaluators.
• Single evaluator ratings yielded lower ICCs (.27-.38).
• Bland-Altman analysis indicated minimal consistent bias and better agreement for higher-rated teachers compared to video, with qualitative themes highlighting the utility and limitations of audio-only data.

Conclusions:

• Audio-only MBI:TAC assessments demonstrate adequate inter-rater reliability for many research and clinical applications.
• Utilizing an average rating from multiple evaluators enhances the reliability of audio-based assessments.
• Audio-only ratings may present greater challenges when assessing less experienced teachers.