Scoring German Alternate Uses Items Applying Large Language Models.

Area of Science:

• Psychology
• Artificial Intelligence
• Cognitive Science

Background:

• The alternate uses task (AUT) is a key measure for assessing creative potential.
• Human scoring of AUT responses is resource-intensive and time-consuming.
• Large language models (LLMs) offer a potential solution for automated AUT scoring.

Purpose of the Study:

• To evaluate the effectiveness of different LLMs for scoring German AUT responses.
• To compare fine-tuned multilingual LLM approaches (CLAUS, OCSAI) with GPT-4.
• To assess the generalizability of LLM performance across diverse datasets and conditions.

Main Methods:

• Compiled a large dataset of ~50,000 German AUT responses from multiple studies.
• Compared OCSAI, CLAUS, and GPT-4 for scoring original German responses and English translations.
• Utilized a pre-registered analysis plan to ensure methodological rigor.

Main Results:

• LLM-based scorings demonstrated substantial correlations with human ratings.
• OCSAI showed the highest correlation, followed by GPT-4 and CLAUS.
• Translating responses to English did not consistently enhance scoring accuracy.

Conclusions:

• LLMs, particularly OCSAI and GPT-4, are effective tools for automated scoring of German AUT responses.
• The choice of LLM impacts scoring accuracy, with OCSAI and GPT-4 being strong candidates.
• Further research should explore generalizability and optimize LLM application for creativity assessment.