Large Language Models and the Wisdom of Small Crowds

Area of Science:

• Computational Linguistics
• Psycholinguistics
• Artificial Intelligence

Background:

• Large Language Models (LLMs) are increasingly used in research, raising questions about their ability to replace human-generated data.
• The hypothesis that LLMs capture collective human knowledge (
• wisdom of the crowd
• ) from vast training data lacks robust empirical validation.

Purpose of the Study:

• To introduce and validate a novel methodological framework, the "number needed to beat" (NNB), for comparing the quality of human data against LLM-generated data.
• To assess the utility of the NNB method across diverse psycholinguistic datasets.
• To explore hybrid approaches combining LLM and human data.

Main Methods:

• Development of the "number needed to beat" (NNB) metric to quantify the human sample size required to match LLM (GPT-4) performance.
• Collection of novel human data across four English psycholinguistic datasets.
• Implementation and evaluation of two "centaur" methods integrating LLM and human data.

Main Results:

• The NNB was greater than 1 for all tested psycholinguistic datasets, indicating human data still offers unique value.
• NNB varied across tasks, with some requiring only a small number of human participants (e.g., 2) to rival LLM quality.
• Hybrid "centaur" approaches combining LLM and human data demonstrated superior performance compared to either data source alone.

Conclusions:

• The NNB framework provides a quantitative method for evaluating the integration of LLM-generated data into research workflows.
• While LLMs offer potential, human data remains crucial, and hybrid approaches can optimize data quality and cost-effectiveness.
• This framework can guide researchers in making informed decisions about leveraging LLM data alongside traditional human subject research.