When 'Yes' Meets 'But': Can AI Comprehend Contradictory Humor in Comics?

Area of Science:

• Artificial Intelligence
• Cognitive Science
• Computational Linguistics

Background:

• Large vision language models (VLMs) face challenges in understanding complex humor requiring comparative reasoning.
• This limitation impacts AI's capacity for human-like reasoning and cultural expression.

Purpose of the Study:

• To analyze humor in comics that use contradictory juxtapositions.
• To introduce and utilize the YESBUT benchmark for evaluating VLMs' narrative and comparative reasoning abilities.
• To identify specific weaknesses in current VLM performance.

Main Methods:

• Development of the YESBUT benchmark (1,262 comic images) with diverse contexts and annotations.
• Systematic evaluation of various VLMs on four tasks, focusing on comparative reasoning.
• Investigation of text-based training and social knowledge augmentation strategies.

Main Results:

• Advanced VLMs significantly underperform compared to human capabilities in understanding comic humor.
• Common failure points include visual perception, key element identification, comparative analysis, and hallucinations.
• Current models lack deep narrative understanding, especially with contradictory elements.

Conclusions:

• VLMs exhibit critical weaknesses in understanding cultural and creative expressions, particularly humor based on contradiction.
• The YESBUT benchmark provides a pathway for assessing and improving VLM narrative reasoning.
• Future research should focus on developing context-aware models with enhanced comparative reasoning skills.