More than just noise: careless responding and its systematic effects on reliability, validity, and measurement...

Careless responding, in which survey participants fail to attend to item content, is a well-recognized threat to the quality of self-report data. Although prevalence estimates commonly fall between 8 and 12% in student samples, the extent to which careless responding distorts the psychometric properties of attitude measures has received limited attention, particularly outside Western samples.

The present study investigated the prevalence and psychometric consequences of careless responding in a paper-and-pencil sample of 1,112 Turkish university students who completed the sustainable development awareness scale. An instructed response item embedded within the scale identified 126 respondents (11.33%) as careless, and two post-hoc indicators (longstring and even-odd consistency) converged with this classification. Parallel analyses on the unscreened and screened samples were conducted to evaluate effects on internal consistency, confirmatory factor analysis, multigroup measurement invariance, criterion-related validity correlations, and an item-level Composite Sensitivity Index (CSI).

Internal consistency was higher in the screened sample, with the gains concentrated in the two subscales containing reverse-coded items. Confirmatory factor analysis indices trended in the direction of better fit after screening, and criterion-validity correlations with related constructs were modestly larger in the screened sample at both the manifest and latent levels. Multigroup measurement invariance testing across attentive and careless responders supported metric but not scalar invariance, pointing to systematic intercept differences consistent with acquiescent responding among careless respondents. The composite sensitivity index combining changes in item means, item-total correlations, and factor loadings was used to rank items by vulnerability to careless responding, with results robust to an alternative standardized aggregation. All six reverse-coded items appeared among the ten most sensitive items, despite constituting only one-sixth of the scale, and the item immediately following the attention check showed a notably elevated sensitivity pattern that we interpret as a tentative hypothesis worth further investigation.

These findings underscore the importance of routine screening for careless responding and offer practical guidance on the placement of attention checks and the use of reverse-coded items.