Uncovering the semantics of concepts using GPT-4

Area of Science:

• Computational Social Science
• Natural Language Processing
• Artificial Intelligence

Background:

• Recent Large Language Models (LLMs) demonstrate human-like text generation capabilities.
• LLMs offer potential for creating semantic similarity measures aligned with human judgment.
• Typicality, the similarity of a text to a concept, is a key semantic measure.

Purpose of the Study:

• To empirically test the utility of LLMs for constructing human-aligned semantic typicality measures.
• To evaluate GPT-4's performance in generating a typicality measure against existing benchmarks.
• To assess the zero-shot learning capabilities of LLMs for social science applications.

Main Methods:

• Utilized GPT-4 to develop a novel typicality measure.
• Compared the GPT-4 typicality measure against prior state-of-the-art model-based measures.
• Evaluated performance using correlations with human typicality ratings in two distinct datasets: book descriptions within literary genres and US Congress members' tweets.

Main Results:

• The GPT-4 based typicality measure met or exceeded the performance of the previous state-of-the-art measure.
• This high performance was achieved using zero-shot learning, without any training on the research data.
• The previous state-of-the-art measure required extensive fine-tuning of a Large Language Model.

Conclusions:

• GPT-4 can effectively generate a measure of text typicality that aligns with human judgment.
• Zero-shot learning with advanced LLMs presents a significant advancement over previous fine-tuning approaches for semantic analysis in social sciences.
• This research validates LLMs as a powerful, efficient tool for social scientists to construct semantic similarity measures.