Assessing inconsistent responding in digital and paper formats: a replication study using the TIMSS 2023 Confidence...

Area of Science:

• Educational measurement
• Psychometrics
• Survey methodology

Background:

• Inconsistent responding in surveys can bias results.
• Mixed-worded scales present unique challenges for respondents.
• Digital administration is increasingly common in large-scale assessments.

Purpose of the Study:

• To examine inconsistent responding in the TIMSS 2023 Confidence in Mathematics Scale.
• To compare digital versus paper questionnaire administrations.
• To identify predictors of inconsistent responding.

Main Methods:

• Posttest-only experimental design comparing digital and paper formats.
• Utilized Mean Absolute Difference and Factor Mixture Analysis for detection.
• Employed binary logistic regression to analyze student characteristics.

Main Results:

• Digital administration showed slightly higher inconsistent responding rates than paper.
• Fourth-grade students had higher rates of inconsistent responding than eighth-grade students.
• Mathematics achievement was the strongest negative predictor of inconsistent responding.

Conclusions:

• Administration mode impacts response consistency in large-scale assessments.
• Digital testing requires careful consideration to mitigate response bias.
• Understanding student characteristics, like math achievement, is crucial for improving survey data quality.