Comprehending how visual context influences incremental sentence processing: Insights from ERPs and picture-sentence...

Area of Science:

• Cognitive Neuroscience
• Psycholinguistics
• Computational Linguistics

Background:

• Picture-sentence verification tasks are used to study language comprehension.
• Previous studies relied heavily on response time (RT) measures, potentially overlooking early processing stages.
• The role of situated comprehension and early neural markers in verification tasks requires further investigation.

Purpose of the Study:

• To re-establish picture-sentence verification as a valid task for situated comprehension research.
• To investigate the temporal dynamics of verification processing using event-related brain potentials (ERPs).
• To explore the relationship between neural activity, behavioral responses, and working memory capacity.

Main Methods:

• Collected event-related brain potentials (ERPs) and response times (RTs) while participants verified sentences against depicted actions.
• Analyzed ERPs, focusing on the N400 component to verbs and subsequent object noun processing.
• Correlated ERP measures and RT effects with individual verbal working memory scores.

Main Results:

• Mismatches between sentences and pictures elicited longer RTs and larger centro-parietal N400s to verbs.
• ERPs to object nouns were more negative for mismatches.
• RT effects correlated inversely with verb N400s and positively with object ERP congruence effects.
• Verb N400s, object ERPs, and working memory scores together explained 50% of the variance in RT effects.

Conclusions:

• Sentence verification involves processing that occurs during, not solely after, sentence reading.
• The observed effects cannot be explained by simple priming mechanisms.
• Verification tasks, when incorporating neural measures, can provide valuable insights into situated language comprehension.