Evaluating large language models in analysing classroom dialogue

Area of Science:

• Educational Technology
• Artificial Intelligence in Education
• Natural Language Processing

Background:

• Traditional analysis of classroom dialogue for teaching diagnosis is time-consuming and labor-intensive.
• There is a need for more efficient and scalable methods to evaluate teaching quality through dialogue analysis.

Purpose of the Study:

• To investigate the efficacy of Large Language Models (LLMs), specifically GPT-4, in analyzing classroom dialogue.
• To compare LLM-based analysis with traditional manual coding methods for accuracy and efficiency.

Main Methods:

• Classroom dialogues from middle school mathematics and Chinese classes were collected.
• Expert human coders manually annotated the dialogue data.
• A customized GPT-4 model was developed and used to analyze the same dialogue datasets.
• Manual annotations were compared against GPT-4 outputs using metrics such as time efficiency and inter-coder agreement.

Main Results:

• LLM analysis demonstrated significant time savings compared to manual coding.
• High coding consistency and reliability were observed between GPT-4 outputs and human coders.
• Minor discrepancies between the model and human coders were noted, requiring further investigation.

Conclusions:

• LLMs, particularly GPT-4, show strong potential for streamlining the analysis of classroom dialogue.
• AI-powered tools can enhance teaching evaluation and facilitate quality improvement in educational settings.
• Further research is warranted to refine LLM customization for nuanced educational contexts.