Using Mokken scaling techniques to explore carelessness in survey research

Area of Science:

• Psychometrics
• Survey Methodology
• Data Quality Assurance

Background:

• Careless responding is a significant challenge in survey research, potentially compromising data validity.
• Existing methods for detecting carelessness include summary statistics and parametric item response theory (IRT) models.
• Nonparametric IRT methods, specifically Mokken scale analysis (MSA), have been underutilized for identifying careless survey responses.

Purpose of the Study:

• To investigate the sensitivity of Mokken scale analysis (MSA) indicators to examinee carelessness in survey research.
• To compare the performance of MSA indicators against standalone carelessness detection methods.
• To assess the impact of participant carelessness on the quality indicators within MSA.

Main Methods:

• Utilized a real data analysis and a simulation study to evaluate MSA's effectiveness.
• Examined both numeric and graphical indicators of response quality derived from MSA.
• Compared MSA indicators with traditional carelessness detection techniques.

Main Results:

• MSA's numeric and graphical indicators of response quality demonstrated sensitivity to examinee carelessness.
• Graphical displays of nonparametric person response functions (PRFs) offered valuable supplementary insights into problematic response patterns.
• MSA's item quality indicators proved robust, showing resilience to the presence of participant carelessness.

Conclusions:

• Mokken scale analysis (MSA) provides a valuable framework for detecting careless responding in survey research.
• Nonparametric person response functions (PRFs) within MSA can enhance the identification of response quality issues.
• MSA offers a reliable approach to maintaining data quality, even in the presence of careless participants.