Investigating Bodily Responses to Unknown Words: a Focus on Facial Expressions and EEG

Occurrences of unknown words in a conversation can be challenging and often prevent people from engaging in fluent communication with each other. Even worse, currently very little is known about possible bodily responses when a listener comes across unknown words, especially when context information is not available in the conversation to facilitate understanding. In this work, we look at facial expressions and electroencephalography (EEG) as two potential body signals that may convey whether users are having difficulties understanding the words they hear. We performed an experiment to measure the reaction of users during a vocabulary dictation test using meaningful words and pseudowords. Participants were asked to classify words as they heard them into different categories. As a result, we did not see any significant differences in the facial expressions of our participants. However, significant differences were observed in event-related potentials (ERPs) within the time range of 100ms-300ms since the onset of stimuli, with pseudowords showing significantly stronger negative responses than meaningful words. Starting at about 550ms and up to around 750ms, pseudowords elicited significantly stronger negative responses, primarily over the parietal and central brain regions. Analyses for single-electrode sites revealed that pseudowords elicited more negative responses than real words in all investigated regions except the left temporal and lateral frontal regions from 500ms to 700ms since stimuli onset. These results could pave the way for future work that aims to develop real-time solutions for facilitating communication between users with different language backgrounds.