Understanding lightbulb moments: Meaning-making in visual morphology from comics and emoji

Area of Science:

• Cognitive Psychology
• Visual Semantics
• Human-Computer Interaction

Background:

• Interpreting abstract concepts from visual cues is crucial for communication.
• Upfixes, like lightbulbs or gears above heads, are common visual metaphors.
• Understanding the semantic processing of these upfixes is key to deciphering visual language.

Purpose of the Study:

• To investigate the semantic processing of visual metaphors (upfixes).
• To determine how congruity between facial expressions and upfixes influences meaning interpretation.
• To explore whether upfixes alone can trigger their associated meanings.

Main Methods:

• Participants viewed sequences of facial expressions and upfixes, followed by literal or non-literal words.
• Response times and congruity ratings were collected for matching and mismatching face-upfix dyads.
• A combinatorial method using verbal and visual modalities was employed.

Main Results:

• Literal words showed faster responses when presented before images.
• Non-literal words were processed faster for matching face-upfix dyads when images preceded words.
• Congruity ratings indicated non-literal words matched congruent dyads, while literal words matched incongruent ones.

Conclusions:

• Non-literal meanings of upfixes are recalled effectively only when they align with facial expressions.
• This supports the existence of a constrained visual lexicon for non-literal expressions.
• The study highlights the interplay between visual and verbal modalities in memory for symbolic imagery.