Neural mechanisms of video learning influenced by pedagogical agents' image and voice: an fNIRS study

In multimedia learning environments, pedagogical agents are believed to facilitate learning through social cues. However, further research is needed to determine how to design pedagogical agents to optimize the learning experience and enhance learning outcomes.

This study aims to investigate how the visual and auditory characteristics of pedagogical agents in multimedia learning influence learners' performance and brain activation patterns during the learning process.

A total of 30 participants were recruited to participate in a multimedia course on aerospace-related knowledge. The experiment employed a 2 (image type: human / animation) × 2 (voice type: human / machine) within-subjects design. Functional near-infrared spectroscopy (fNIRS) was used to measure cortical activity during the learning process, and then participants completed learning outcome tests, including knowledge retention and transfer tests.

Regarding learning performance, the retention test revealed a significant interaction between visual and auditory modalities, with the combination of animated agents and human voices yielding the best results; the transfer test showed a significant main effect of voice type, with human voices significantly outperforming machine voices. fNIRS results indicated that, compared to human agents, animated agents elicited higher levels of brain activation in regions such as the inferior frontal gyrus, middle temporal gyrus, frontal pole, and dorsolateral prefrontal cortex; compared to machine voices, human voices elicited stronger activation responses in regions such as the frontal pole, middle temporal gyrus, and dorsolateral prefrontal cortex. The combination of animated agents and human voices exhibited the strongest brain activation pattern, suggesting that multimodal social cues may promote higher levels of cross-modal integration processing under these conditions.

This study reveals the mechanisms underlying the synergistic effects of a pedagogical agent's image and voice at the behavioral and cognitive-neural levels, providing empirical evidence for the optimized design of pedagogical agents in multimedia learning environments.