A change-point analysis procedure based on weighted residuals to detect back random responding.

Area of Science:

• Psychometrics
• Statistical Methods

Background:

• Careless responding, particularly back random responding (BRR), compromises the validity and generalizability of questionnaire and survey data.
• Existing methods for detecting BRR often have low power (around 0.5 or lower), making them unreliable.
• Change Point Analysis (CPA) is a statistical process control method used to detect aberrant response patterns, but traditional CPA may not be suitable for non-directional changes like BRR.

Purpose of the Study:

• To propose and evaluate a novel weighted-residual-based Change Point Analysis (CPA) procedure for detecting back random responding (BRR) in survey data.
• To compare the performance of the proposed method against existing CPA techniques in terms of detection power and Type-I error rate.
• To demonstrate the practical utility of the new method using a real-world dataset.

Main Methods:

• Development of a weighted-residual-based Change Point Analysis (CPA) procedure tailored for detecting non-directional response behaviors like BRR.
• Comprehensive simulation studies to evaluate the proposed CPA method's performance, including detection power and Type-I error rates.
• Comparison of the proposed method with three existing CPA methods using simulation data and an empirical dataset.

Main Results:

• The proposed weighted-residual-based CPA procedure demonstrated high power in detecting BRR for tests with 20 or more items, while maintaining well-controlled Type-I error rates.
• The new method achieved comparable empirical Type-I error rates to existing CPA methods but offered a significant gain in detection power (17%-42%).
• An empirical study confirmed the practical effectiveness of the proposed method in identifying BRR in actual survey data.

Conclusions:

• The weighted-residual-based CPA is a powerful and effective tool for detecting back random responding (BRR) in psychometric and survey research.
• This method offers a significant improvement over existing techniques, enhancing data quality and research findings.
• Further research should explore the method's limitations and potential applications in diverse research contexts.