A generalizability study of student ratings in logbooks to assess the educational process of clinical learning

To investigate the number of student raters necessary for a reliable estimate of how educators approach the process of clinical learning (i.e., levels of student participation and educator feedback) in clinical settings.

Third-year medical students documented clinical experiences across seven core clerkships by using an online system. The authors asked students to identify their clinical encounters and to respond to questions about the educational process. The authors' hypothesis was that educators would have different preferences with regard to students' participation in the clinical experience, as well as different preferences with regard to offering their own feedback to students. The authors conducted a generalizability study to ascertain the variance components for the responses to the educator feedback and the student participation log questions. This generalizability study design was a rater-nested-with-person design, noted as r:p. The authors also conducted a decision study to calculate projected G (generalizability) coefficients across different numbers of raters. The results can help determine the minimum number of raters required to reach a reliable estimate of whether a faculty member offers feedback and invites students to participate in the clinical encounter.

The generalizability study found that most of the variance components were attributed to educators. The decision study found that the G coefficients reached acceptable levels of reliability when at least five raters completed clinical encounters with an educator.

The results suggest that students' responses reliably distinguished the levels of educator feedback and encouragement that were needed for student participation in clinical encounters.