Revealing teaching quality through lesson semantics: A GPT-assisted analysis of transcripts

Existing conceptions of teaching quality assume that classroom interactions serve as the foundation for effective teaching. The resulting data necessitates analytical approaches capable of extracting the semantics of these interactions.

This study investigates whether and to what extent lesson semantics provide insights into teaching quality (i.e., cognitive engagement, encouragement and warmth, multiple approaches, and the nature of discourse). To achieve this, GPT-4 was applied as a tool for analysing lesson transcripts.

The study is based on data from the TALIS Video study, which included N = 50 teachers delivering two consecutive mathematics lessons in 9th grade. Teaching quality was annotated by trained observers across multiple dimensions.

The analysis involved embedding segmented lesson transcripts to examine their semantic characteristics and associations with human annotations of teaching quality. Additionally, we applied content-informed prompting to evaluate the interpretability of semantic characteristics for the considered dimensions.

GPT-4 identified five distinct semantic representations of transcripts, varying at both the teacher and lesson levels. These representations were related to teaching quality, accounting for up to 20% of variance in teaching quality annotations. Content-informed prompting aligned lesson segments more closely with semantic representations, supporting their interpretability.

The findings suggest that lesson semantics serve as indicators of teaching quality, offering a promising approach to understanding effective classroom learning.