Comparing Person-Fit and Traditional Indices Across Careless Response Patterns in Surveys.

Area of Science:

• Psychometrics
• Survey Methodology
• Educational Measurement

Background:

• Careless responding in surveys introduces bias, necessitating robust identification methods.
• Multiple indices are typically employed due to the varied nature of carelessness.
• Item-response theory (IRT) offers advanced statistical frameworks for data analysis.

Purpose of the Study:

• To evaluate the efficacy of three IRT-based person-fit indices (infit MSE, outfit MSE, and the polytomous l statistic) in identifying careless survey responses.
• To compare the performance of these IRT indices against traditional carelessness detection methods.
• To assess the bidirectional capability of person-fit indices in detecting both random and overconsistent response patterns.

Main Methods:

• Utilized empirical data from 2,049 high school student surveys on teaching effectiveness.
• Employed simulated data with manipulated types (random, overconsistent) and percentages of carelessness.
• Compared IRT person-fit statistics (infit MSE, outfit MSE, l statistic) with traditional indices (LongString, Mahalanobis Distance, Validity Items, Completion Time).

Main Results:

• IRT-based person-fit indices (infit MSE, outfit MSE, l statistic) demonstrated complementary value to traditional carelessness indices.
• Receiver operating characteristic (ROC) curves indicated good sensitivity and specificity for person-fit indices in detecting both over-consistent and under-consistent (random) careless patterns.
• Classifications from person-fit indices showed correlations with classifications from traditional methods, suggesting a bidirectional detection capability.

Conclusions:

• IRT-based person-fit indices offer a valuable, bidirectional approach to identifying careless responding in surveys.
• These indices can enhance the validity of survey data by complementing existing detection techniques.
• Findings have implications for improving survey development, validation, and data interpretation in research and practice.