Employing large language models for cognitive heuristic stimulus validation in decision-making experiments

This study explores the use of large language models (LLMs), specifically ChatGPT, as a tool for validating cognitive heuristic stimuli for decision-making experiments. Traditional validation methods rely on human judges to ensure that experimental stimuli reflect intended heuristics, but they face challenges related to bias, resource demands, and interpretative consistency. Here, ChatGPT classifies decision-making rationales into 10 heuristics, including affect, availability, and framing. By employing iterative refinement and chain-of-thought prompting, we identify and address heuristic misalignment and ambiguity in experimental rationales, improving both precision and accuracy in heuristic labeling. Results demonstrate the effectiveness of LLMs in achieving alignment with intended heuristic categories while significantly reducing validation time and resources. The findings highlight the potential of LLMs as a cost-effective, scalable alternative to traditional validation techniques, with implications for future research in cognitive science and artificial intelligence (AI)-supported decision-making systems. Although LLMs are potent tools for stimulus validation, successful deployment depends on careful model selection, understanding of variability, and precise contextualization, especially in complex domains. Future experiments leveraging the validated stimuli will aim to assess preferences for heuristics relative to rational reasoning and inform the design of decision-support tools.