Development and initial validation of a video-mediated listening test based on multimodal generative AI.

Area of Science:

• Educational Technology
• Artificial Intelligence in Education
• Language Assessment

Background:

• Video-mediated listening offers rich contextual cues for improved comprehension and engagement.
• Developing authentic listening assessments is resource-intensive, limiting practical application.
• Generative AI presents a potential solution to these development challenges.

Purpose of the Study:

• To apply generative AI for creating a video-mediated listening assessment.
• To evaluate the validity and utility of an AI-generated listening assessment.
• To explore the impact of AI-driven video-mediated listening on student learning and teacher workload.

Main Methods:

• Utilized multimodal generative AI (InVideo) for video creation and GPT-4 for test item generation.
• Administered the AI-based assessment to 542 university students, followed by questionnaires.
• Conducted semi-structured interviews with five students and five teachers for qualitative insights.

Main Results:

• Psychometric analysis confirmed 19 out of 20 items effectively measured listening subskills with construct validity.
• Student questionnaires indicated AI-generated videos were adequate and supported test relevance.
• Qualitative data revealed enhanced student interest, motivation, and engagement, and reduced teacher preparation time.

Conclusions:

• AI-generated video-mediated listening assessments are valid and effective tools for language learning.
• This AI approach enhances student engagement and motivation compared to traditional audio methods.
• Multimodal AI offers practical benefits for language educators by reducing material preparation burdens.