Quantifying teaching quality in medical education: The impact of learning gain calculation

Student performance is a mirror of teaching quality. The pre-/post-test design allows a pragmatic approach to comparing the effects of interventions. However, the calculation of current knowledge gain scores introduces varying degrees of distortion. Here we present a new metric employing a linear weighting coefficient to reduce skewness on outcome interpretation.

We compared and contrasted a number of common scores (raw and relative gain scores) with our new method on two datasets, one simulated and the other empirical from a previous intervention study (n = 180) employing a pre-/post-test design.

The outcomes of the common scores were clearly different, demonstrating a significant dependency on pre-test scores. Only the new metric revealed a linear relationship to the knowledge baseline, was less skewed on the upper or lower extremes, and proved well suited to allow the calculation of negative learning gains. Employing the empirical dataset, the new method also confirmed the interaction effect of teaching formats with specific subgroups of learner characteristics.

This work introduces a new weighted metric enabling meaningful comparisons between interventions based on a linear transformation. This method will form the basis to intertwine the calculation of test performance closely with the outcome of learning as an important factor reflecting teaching quality and efficacy. Its regular use can improve the transparency of teaching activities and outcomes, contribute to forming rounded judgements of students' acquisition of knowledge and skills and enable valuable feedforward to develop and enhance curricular concepts.