The Two Word Test as a semantic benchmark for large language models.

Area of Science:

• Natural Language Processing
• Artificial Intelligence
• Cognitive Science

Background:

• Large language models (LLMs) demonstrate advanced capabilities, prompting discussions about their potential for human-like understanding and artificial general intelligence (AGI).
• Current benchmarks often focus on reasoning or domain expertise, potentially overlooking fundamental semantic processing abilities.
• Human language relies heavily on combining words to form meaningful concepts, a core linguistic operation.

Purpose of the Study:

• Introduce the open-source Two Word Test (TWT) as a novel benchmark to assess the semantic abilities of LLMs.
• Evaluate LLMs' capacity for meaningfulness judgments on two-word phrases, a task easily performed by humans.
• Provide a tool to identify and address limitations in LLM language understanding.

Main Methods:

• Developed the Two Word Test (TWT), comprising 1768 noun-noun combinations rated for meaningfulness by human participants.
• Administered TWT in two versions: a 0-4 scale for nuanced ratings and a binary judgment task.
• Tested leading LLMs including GPT-4, GPT-3.5, Claude-3-Optus, and Gemini-1.0-Pro-001 on the TWT.

Main Results:

• All tested LLMs performed significantly worse than humans in judging the meaningfulness of two-word phrases.
• GPT-3.5-turbo, Gemini-1.0-Pro-001, and GPT-4-turbo failed to reliably distinguish between meaningful and nonsensical phrases.
• Claude-3-Opus showed improvement in binary discrimination but still lagged behind human performance.

Conclusions:

• The TWT effectively highlights the limitations of current LLMs in fundamental semantic understanding.
• Results suggest caution is needed when attributing human-level or "true" understanding to LLMs based on existing benchmarks.
• The TWT offers a valuable tool for assessing and potentially enhancing the semantic capabilities of future LLMs.