Developing a quality assessment rubric for online videos used for teaching procedural skills

The quality of online medical videos is often unknown despite their common use in health professional education. A number of studies have shown that online videos, especially on YouTube, are of variable quality. The aim of this mixed methods study is to describe the conceptual design, item development and content validity of a rubric constructed to assess the quality of online procedural videos.

We used a multi-stage mixed methods design with a pragmatist and inductive approach over the years 2023-2024. The study had three phases: 1) rubric development phase, 2) content validity of items and scale assessment, and 3) rubric revision. Health professional educators (academic staff) and learners (postgraduate students) were recruited, and the rubric items were developed using a modified Delphi process in two rounds: an anonymized survey and two facilitated online focus groups. In the second phase, some of the focus group participants rated rubric items for clarity and content. Content validity indices (CVI) were calculated. Finally, rubric items with I-CVI (item-level CVI) <0.78 were modified based on feedback.

The responses from 32 participants (17 educators, 15 learners) were analyzed, and nine questions met consensus in the rubric item development stage. Through the Delphi process, a preliminary 18-item rubric was created. 94% of items had an I-CVI of ≥ 0.78 for clarity. One item received an I-CVI of 0.75. The S-CVI/average of the questionnaire achieved an acceptable level of 0.93 for content clarity. 100% of items had an I-CVI of ≥ 0.78. The S-CVI/questionnaire average (scale-level CVI/average) was an acceptable level of 0.91 for relevance.

We have developed a rubric to assess the quality of online videos used to teach procedural skills. It has good content validity. Further analysis is needed to determine its psychometric properties, feasibility, and user experience.