Random Forest Predicts Human Ratings of Creative Stories Using Very Small Training Samples

Area of Science:

• Psychology
• Artificial Intelligence
• Computational Social Science

Background:

• The Consensual Assessment Technique (CAT) is a validated method for assessing creativity.
• CAT implementation faces challenges due to rater burden and complex designs.

Purpose of the Study:

• To investigate machine learning's ability to simulate expert creativity judgments.
• To identify optimal algorithms and minimal training data for predicting CAT scores.

Main Methods:

• Utilized a dataset of 411 short stories.
• Compared Random Forest, Gradient Boosted Trees, and Decision Tree models.
• Used story length and Divergent Semantic Integration as features to predict CAT ratings.

Main Results:

• Random Forest (RF) achieved high correlation (r = 0.80) with CAT scores.
• RF required only 25 training stories for reliable prediction.
• RF outperformed other algorithms and LLM-based models in accuracy and reduced reliance on story length.

Conclusions:

• Machine learning, particularly RF, offers a scalable and efficient method for simulating expert creativity assessment.
• This approach reduces human rater burden and logistical complexities.
• Further research is needed to assess generalizability across different creative domains.