Evaluating Large Language Models' Ability Using a Psychiatric Screening Tool Based on Metaphor and Sarcasm Scenarios

Area of Science:

• Artificial Intelligence
• Cognitive Science
• Linguistics

Background:

• Metaphor and sarcasm are key aspects of human social communication.
• Children with Asperger syndrome often exhibit difficulties comprehending sarcasm.
• Previous research used metaphor and sarcasm comprehension tests to differentiate Asperger syndrome.

Purpose of the Study:

• To evaluate large language models' (LLMs) comprehension of nuanced human communication, specifically metaphors and sarcasm.
• To assess the impact of model parameters on LLMs' ability to understand these complex linguistic elements.

Main Methods:

• Utilized a standardized test to assess metaphor and sarcasm comprehension in recent LLMs.
• Analyzed the relationship between LLM size (parameters) and performance on both comprehension tasks.

Main Results:

• LLMs demonstrated improved metaphor comprehension correlating with increased model parameters.
• No significant improvement in sarcasm comprehension was observed with larger model sizes.
• LLM performance on sarcasm did not mirror the pattern seen in human development.

Conclusions:

• Current LLM architectures may require distinct training strategies to develop robust sarcasm comprehension.
• Cognitively grounded approaches, potentially inspired by human emotional learning centers like the amygdala, are necessary for AI to grasp sarcasm.
• Further research is needed to bridge the gap in LLMs' understanding of subtle social communication cues.