Detecting Careless Responding in Survey Data Using Stochastic Gradient Boosting.

Area of Science:

• Psychological Measurement
• Survey Methodology
• Machine Learning Applications

Background:

• Careless responding poses a significant threat to the reliability and validity of psychological measurements by disregarding item content.
• Existing methods for detecting aberrant responses include probing questions, paradata (e.g., response times), and statistical techniques (e.g., Mahalanobis distance).

Purpose of the Study:

• To introduce gradient boosted trees, a machine learning technique, for identifying careless respondents in survey data.
• To compare the performance of gradient boosting machines against established detection methods using simulated and empirical data.

Main Methods:

• Gradient boosted trees were employed as a novel machine learning approach to detect careless responding.
• Performance was evaluated against traditional methods (outlier methods, consistency analyses, response pattern functions).
• Both simulated data and empirical data from an experimentally induced careless responding study were utilized.

Main Results:

• In simulation studies, gradient boosting machines demonstrated superior performance in flagging aberrant responses compared to traditional methods.
• This performance advantage did not translate to the empirical study; precision was unsatisfactory for both novel and traditional methods.
• Real-world survey responses appeared more erratic than anticipated by simulation studies, impacting detection accuracy.

Conclusions:

• The effectiveness of gradient boosting machines for detecting careless responding is promising in simulations but requires further validation in real-world settings.
• Current detection methods, both traditional and novel, exhibit limitations in precision for identifying aberrant response patterns.
• Future research should focus on improving the generalizability and accuracy of detection methods for real-world survey data.