Can large language models help augment English psycholinguistic datasets?

Area of Science:

• Cognitive Science
• Computational Linguistics
• Psycholinguistics

Background:

• Psycholinguistic datasets, or norms, are crucial for language and cognition research.
• Collecting human judgments for these norms is time-consuming and expensive, especially for large-scale or multi-dimensional data.
• Large language models (LLMs) present a potential solution to overcome these data collection challenges.

Purpose of the Study:

• To investigate the feasibility of using LLMs, specifically GPT-4, to generate psycholinguistic norms for English.
• To compare LLM-generated semantic judgments against human-generated data (the gold standard).
• To assess the utility of LLM-generated norms in statistical modeling and identify potential limitations.

Main Methods:

• GPT-4 was employed to collect various semantic judgments (e.g., word similarity, contextualized sensorimotor associations, iconicity) for English words.
• LLM-generated judgments were systematically compared against established human-generated norms.
• Substitution analyses were conducted, replacing human norms with LLM norms in statistical models.

Main Results:

• GPT-4 judgments showed positive correlations with human judgments across different datasets.
• In some cases, LLM agreement levels met or exceeded average human inter-annotator agreement.
• Substitution analyses indicated that LLM norms generally preserve the direction of parameter estimates in statistical models, though magnitudes may change.

Conclusions:

• LLMs can effectively augment the creation of large-scale psycholinguistic datasets, offering a cost-effective and efficient alternative to human data collection.
• Systematic differences exist between LLM and human norms, requiring careful consideration of factors like data contamination, LLM choice, and validity.
• The study provides valuable LLM-generated norm data and discusses critical considerations for their future use in research.