ERP correlates of semantic inconsistencies in deepfakes

Area of Science:

• Neuroscience
• Cognitive Science
• Media Studies

Background:

• Deepfake technology advances pose challenges for distinguishing real from fake media.
• Understanding the cognitive processing of deepfakes, especially concerning sensitive topics like vaccination, is crucial.

Purpose of the Study:

• To investigate how speaker credibility and participant attitudes influence brain responses to audio deepfakes about COVID-19 vaccination.
• To explore the neural mechanisms underlying the processing of deepfake content based on source credibility and message congruence.

Main Methods:

• Electroencephalograms (EEGs) were recorded from 61 participants with varying stances on COVID-19 vaccination.
• Participants listened to audio deepfakes featuring a pro-vaccine doctor and an anti-vaccine actress presenting congruent and incongruent statements.
• Neural responses, specifically the N400 component, were analyzed in relation to critical words within the deepfakes.

Main Results:

• Contradictory statements from the credible pro-vaccine doctor elicited a stronger N400-like response compared to congruent statements.
• No similar effect was observed for the anti-vaccine actress, suggesting a role for perceived speaker credibility.
• N400 amplitude to contradictory statements predicted perceived speaker credibility, while responses to congruent statements predicted intentions to share deepfakes.

Conclusions:

• Speaker credibility significantly modulates the neural processing of semantic incongruity in audio deepfakes.
• Brain mechanisms of semantic processing are linked to behavioral findings on source credibility and deepfake sharing intentions.
• This study provides insights into the cognitive and neural underpinnings of deepfake perception and its relation to public health discourse.