Automating creativity assessment with SemDis: An open platform for computing semantic distance

Area of Science:

• Psychology
• Computational Linguistics
• Artificial Intelligence

Background:

• Creativity research traditionally relies on human raters to assess idea quality, posing challenges in labor cost and inter-rater reliability.
• Subjective scoring methods in creativity tasks like the Alternate Uses Task (AUT) face psychometric threats due to rater variability.
• Automated scoring offers a potential solution to enhance the efficiency and objectivity of creativity assessment.

Purpose of the Study:

• To evaluate the efficacy of automated scoring using semantic distance for assessing verbal creativity.
• To compare the performance of various semantic models in predicting human creativity and novelty judgments.
• To address the limitations of subjective scoring in creativity research through computational methods.

Main Methods:

• Utilized five top-performing semantic models (e.g., GloVe, continuous bag of words) to compute semantic distance between texts.
• Assessed semantic models against human creativity ratings from the Alternate Uses Task (AUT) and word association tasks.
• Developed a latent semantic distance factor from common variance across semantic models.

Main Results:

• A latent semantic distance factor strongly and reliably predicted human creativity and novelty ratings across multiple tasks.
• The computational method demonstrated convergent validity by correlating with other creativity measures.
• An established experimental effect (serial order effect) was replicated using semantic distance, supporting its utility.

Conclusions:

• Automated scoring via semantic distance provides a reliable and valid method for assessing verbal creativity.
• Computational approaches can overcome the labor and subjectivity limitations inherent in human scoring of creative outputs.
• An open platform for computing semantic distance is provided to facilitate further research in automated creativity assessment.