Related Experiment Videos
Generative psychometrics via AI-GENIE: Automatic item generation and validation with network-integrated evaluation.
Lara L Russell-Lasalandra1, Alexander P Christensen2, Hudson Golino3
1University of Virginia, Charlottesville, VA, 22904, USA.
Behavior Research Methods
|July 1, 2026
Summary
Artificial intelligence (AI) and large language models (LLMs) now streamline psychological scale development. The AI-GENIE method efficiently generates and validates assessment items, achieving structural validity comparable to traditional methods.
Area of Science:
- Psychological assessment
- Artificial intelligence
- Psychometrics
Background:
- Psychological scale development traditionally relies on expert input and can be time-consuming.
- Advancements in artificial intelligence (AI), particularly large language models (LLMs), offer potential for automating and improving research methodologies.
- Network psychometrics provides advanced techniques for understanding the structure of psychological constructs.
Purpose of the Study:
- To introduce a novel methodology, Automatic Item Generation and Validation with Network-Integrated Evaluation (AI-GENIE), for efficient psychological scale development.
- To leverage LLMs and network psychometrics to automate item generation and selection, reducing reliance on expert judgment.
- To evaluate the efficacy and structural validity of AI-GENIE-generated scales compared to traditional measures.
Main Methods:
- Utilized Monte Carlo simulations to assess AI-GENIE with various LLMs (Mixtral, Gemma 2, Llama 3, GPT-3.5, GPT-4o) for generating Big Five personality items.
- Empirically tested AI-GENIE-generated items with five large, nationally representative U.S. samples (N = 4,964).
- Assessed structural validity, including dimensionality and item stability, of the generated scales.
Main Results:
- AI-GENIE-generated scales demonstrated structural validity comparable to expert-developed measures.
- Significant improvements in item selection efficiency were observed, with normalized mutual information increases of 8.68-20.03.
- AI-GENIE proved effective for developing scales for emerging constructs, such as AI anxiety, using new LLMs.
Conclusions:
- AI-GENIE offers a streamlined and efficient approach to psychological scale development and structural validation.
- The integration of generative AI and network psychometrics represents a significant advancement in assessment methodology.
- This approach holds promise for accelerating the creation of high-quality psychological measures across diverse research areas.
Related Concept Videos
01:16Non-equilibrium in the Cell
An important concept in studying metabolism and energy is that of chemical equilibrium. Most chemical reactions are reversible. They can proceed in both directions, releasing energy into their environment in one direction, and absorbing it from the environment in the other direction. The same is true for the chemical reactions involved in cell metabolism, such as the breaking down and building up of proteins into and from individual amino acids, respectively. Reactants within a closed system...
01:28Automatic Processing and Automatic Social Behavior
Automatic processing refers to the cognitive operations that occur without conscious intent or awareness, playing a fundamental role in shaping social cognition and behavior. These processes enable individuals to navigate complex social environments efficiently by relying on mental shortcuts and pre-existing knowledge structures known as schemas. One of the most influential mechanisms underlying automatic processing is priming, which subtly activates mental representations through exposure to...
01:22Self-Report Tests of Personality
Self-report inventories are objective personality assessments that use multiple-choice items or numbered scales, typically ranging from 1 (strongly disagree) to 5 (strongly agree). They are often called Likert scales after Rensis Likert. These inventories are widely used due to their ease of administration and cost-effectiveness. One of the most prominent examples is the Minnesota Multiphasic Personality Inventory (MMPI), initially developed in the 1940s to assess abnormal personality traits.
01:23Binet's Contribution to Measures of Intelligence
Alfred Binet, along with his student Théophile Simon, was tasked by the French Ministry of Education in 1904 to create a method for identifying students who struggled to learn through conventional classroom instruction. This initiative aimed to address overcrowding by placing such students in specialized schools. Binet and Simon developed an intelligence test comprising 30 tasks, ranging from simple commands, like touching one's nose or ear, to more complex tasks, such as drawing designs from...
01:29Measures of Intelligence
Psychologists measure intelligence by using standardized tests that produce a score known as the intelligence quotient or IQ. To understand IQ tests, it's important to recognize the key principles behind their construction: validity, reliability, and standardization.
Validity refers to how well a test measures what it claims to measure. An intelligence test should accurately assess intelligence rather than another characteristic, like anxiety. Criterion validity is one way to evaluate this; it...
Validity refers to how well a test measures what it claims to measure. An intelligence test should accurately assess intelligence rather than another characteristic, like anxiety. Criterion validity is one way to evaluate this; it...