Let the algorithm speak: How to use neural networks for automatic item generation in psychological scale development

Area of Science:

• Psychological Measurement
• Artificial Intelligence in Research
• Natural Language Processing

Background:

• Developing reliable self-report scales for unobservable psychological constructs is crucial but labor-intensive.
• Traditional scale development requires generating numerous high-quality items, posing a significant challenge for researchers.
• There is a need for efficient, accessible tools to aid in the creation of psychometric scales.

Purpose of the Study:

• To introduce and validate the Psychometric Item Generator (PIG), an open-source natural language processing algorithm.
• To demonstrate the PIG's capability in generating large pools of face-valid items for novel constructs and creating parsimonious short scales for existing ones.
• To provide researchers with an accessible, AI-driven solution for psychometric scale development.

Main Methods:

• Utilized GPT-2, a generative language model, within an open-source, free-to-use algorithm.
• Applied the PIG algorithm through Google Colaboratory, requiring no prior coding skills or computational resources.
• Conducted two demonstrations and a preregistered five-pronged empirical validation with two Canadian samples (N1=501, N2=773).

Main Results:

• The PIG successfully generated large pools of face-valid items for novel constructs (e.g., wanderlust).
• The PIG created parsimonious short scales for existing constructs (e.g., Big Five personality traits) with strong real-world performance.
• Empirical validation confirmed the PIG's effectiveness and benchmarked its output against gold-standard assessment methods.

Conclusions:

• The PIG offers an effective, novel machine learning solution to the challenge of psychometric scale development.
• This tool democratizes scale creation, making advanced AI accessible to researchers without technical expertise.
• The PIG streamlines the generation of customized, human-like text output for diverse research contexts.