Using large language models to estimate belief strength in reasoning.

Area of Science:

• Cognitive Psychology
• Computational Social Science
• Artificial Intelligence

Background:

• Quantifying belief strength in heuristics-and-biases tasks is methodologically challenging.
• Existing methods struggle with systematic measurement and manipulation of belief strength.
• Base-rate neglect tasks, like the lawyer-engineer problem, highlight conflicts between stereotypes and statistical information.

Purpose of the Study:

• To introduce an automated method for measuring and manipulating belief strength using large language models (LLMs).
• To create a comprehensive, open-access database of items varying in stereotype-driven belief strength.
• To validate the LLM-derived belief strength measure and demonstrate its utility in cognitive research.

Main Methods:

• Leveraging large language models (LLMs) to systematically measure and manipulate belief strength.
• Developing over 100,000 unique items for the "lawyer-engineer" base-rate neglect task.
• Creating an open-access database and an R package (baserater) for accessibility.

Main Results:

• LLM-derived belief strength measure shows strong correlation with human typicality ratings.
• The measure robustly predicts human choices in base-rate neglect tasks.
• Identified significant, previously unnoticed variability in stereotype-driven belief strength in existing research items.

Conclusions:

• The LLM-based method offers a powerful, scalable, and precise approach to quantify belief strength.
• The created database and R package facilitate rigorous, replicable cognitive research.
• Methodological refinements and cross-cultural adaptations are possible, advancing cognitive and computational modeling.