Human Expertise and Large Language Model Embeddings in the Content Validity Assessment of Personality Tests

Area of Science:

• Psychometrics and Psychological Assessment
• Artificial Intelligence in Behavioral Science
• Natural Language Processing for Measurement

Background:

• Content validity is crucial for ensuring psychological measures accurately reflect intended constructs.
• Traditional methods rely on human expert judgment, which can be time-consuming and subjective.
• The increasing complexity of psychological constructs necessitates innovative validation approaches.

Purpose of the Study:

• To investigate the application of Large Language Models (LLMs) in assessing the content validity of psychometric instruments.
• To compare the performance of LLMs against human expert evaluations in semantic item-construct alignment.
• To explore the impact of LLM training strategies on content validity assessment accuracy.

Main Methods:

• Human experts (graduate psychology students) used the Content Validity Ratio to evaluate items for the Big Five Questionnaire (BFQ) and Big Five Inventory (BFI).
• State-of-the-art LLMs, including multilingual and fine-tuned models, analyzed item embeddings to predict construct mappings.
• Comparison of semantic item-construct alignment accuracy between human and AI-driven approaches.

Main Results:

• Human validators demonstrated higher accuracy with the behaviorally rich BFQ items.
• LLMs showed superior performance with the linguistically concise BFI items.
• LLM performance was significantly influenced by training strategies, with lexically focused models outperforming general ones.

Conclusions:

• LLMs offer a scalable and objective approach to content validity assessment in psychometric instrument development.
• Hybrid systems integrating human expertise with AI precision present a promising direction for robust test validation.
• LLMs have the potential to transform psychological assessment methodologies, enhancing efficiency and objectivity.