Turing Jest: Distributional Semantics and One-Line Jokes.

Area of Science:

• Cognitive Science
• Artificial Intelligence
• Computational Linguistics

Background:

• Humor is a key human experience, yet its recognition and comprehension remain poorly understood.
• Existing theories focus on incongruity, cognitive shifts, Theory of Mind, and pragmatic reasoning.
• The role of purely linguistic input in humor processing is largely unexplored.

Purpose of the Study:

• To investigate the extent to which large language models (LLMs) can recognize and understand one-line jokes based solely on linguistic data.
• To compare LLM humor comprehension abilities with human performance.
• To identify limitations of distributional approaches to meaning in the context of humor.

Main Methods:

• Multiple preregistered experiments were conducted.
• GPT-3, a large language model (LLM), was tested on joke detection, appreciation, and comprehension tasks.
• Exploratory analysis included open-source LLMs (Llama-3, Mixtral) on similar humor-related tasks.

Main Results:

• GPT-3 exhibited above-chance performance in recognizing and comprehending one-line jokes.
• Other tested LLMs also showed above-chance humor detection and comprehension.
• Both humans and LLMs incorrectly classified surprising non-jokes as humorous.

Conclusions:

• LLMs demonstrate significant, albeit incomplete, capabilities in processing one-line jokes using only language data.
• Current LLM performance falls short of human levels in humor comprehension.
• Findings suggest limitations in purely distributional models for capturing the nuances of humor and meaning.