SEMbeddings: how to evaluate model misfit before data collection using large-language models.

Area of Science:

• Psychometrics
• Computational Linguistics
• Psychological Measurement

Background:

• Large Language Models (LLMs) show potential for approximating empirical correlation matrices using item embeddings and cosine similarities.
• Traditional methods for assessing model fit often occur after data collection, potentially leading to inefficiencies in questionnaire development.

Purpose of the Study:

• Introduce SEMbeddings, a novel tool integrating fine-tuned embedding models with latent measurement models.
• To assess model fit or misfit prior to data collection in psychological measurement.
• To explore the utility of LLM-derived embeddings for informing questionnaire development.

Main Methods:

• SEMbeddings integrates the mpnet-personality model with latent measurement models.
• Applied SEMbeddings to the VIA-IS-P (96 items, 24 character strengths) using responses from 31,697 participants.
• Conducted confirmatory factor analyses on cosine similarity matrices generated by mpnet-personality.

Main Results:

• A significant correlation (r=0.67) was found between embedding cosine similarities and empirical item correlations.
• Traditional fit indices can be misleading with SEMbeddings, suggesting more conservative conclusions.
• Modification indices from SEMbeddings provide valuable insights into potential item misfit.

Conclusions:

• SEMbeddings offers a promising approach for pre-data collection assessment in questionnaire development.
• LLM-derived procedures can enhance the reliability of new questionnaire development.
• Modification indices from SEMbeddings can serve as a screening tool for item selection.