Simulating insufficient effort responding with large language models: A persona-based prompt engineering approach

Area of Science:

• Psychological Science
• Computational Linguistics
• Survey Methodology

Background:

• Traditional methods for simulating insufficient effort responding (IER) data lack ecological validity and controllability.
• Generating realistic IER data is crucial for developing and validating detection methods.
• Large language models (LLMs) offer a novel approach to synthetic data generation.

Purpose of the Study:

• To simulate realistic insufficient effort responding (IER) data using LLMs with persona-based prompt engineering.
• To generate controlled, distinct IER and non-IER datasets for research.
• To improve the ecological validity and controllability of synthetic IER data generation.

Main Methods:

• Utilized persona-based prompt engineering with LLMs to manipulate attributes and generate synthetic IER data.
• Created four persona conditions: one general attentive response and three IER-intended conditions (CRDO, CRPO, CRDP).
• Employed descriptive statistics, correlation analysis, confirmatory factor analysis (CFA), and IER detection indices for validation.

Main Results:

• Persona-based prompt engineering successfully distinguished between IER and non-IER conditions.
• IER-intended conditions exhibited characteristic IER traits, including degraded CFA model RMSEA (CRDO: 0.09, CRPO: 0.13, CRDP: 0.12).
• High mean IER detection rates were observed in IER-intended conditions (CRDO: 52.33%, CRPO: 74.16%, CRDP: 88.66%).

Conclusions:

• Persona-based prompt engineering with LLMs provides a robust and scalable method for simulating realistic human IER.
• This approach generates high-quality synthetic data, enhancing ecological validity and methodological rigor in survey research.
• The methodology lays the groundwork for advancing IER detection accuracy and psychometric validation.