Assessing and Improving Data Integrity in Web-Based Surveys: Comparison of Fraud Detection Systems in a COVID-19...

Area of Science:

• Public Health
• Survey Methodology
• Data Science

Background:

• Web-based surveys enhance participation but face data quality threats like bot entries and duplicate submissions.
• Proprietary fraud detection tools lack transparency in methods and effectiveness.
• Need for robust, reproducible methods to ensure web survey data integrity.

Purpose of the Study:

• Describe a multilayered fraud detection system for a COVID-19 web survey.
• Examine agreement between the custom system and a proprietary system (Qualtrics).
• Compare study samples resulting from each fraud detection method.

Main Methods:

• A cross-sectional web survey (PhillyCEAL Common Survey) assessed COVID-19 impacts.
• Compared a custom multilayer fraud detection strategy (automated validation + human review) with Qualtrics' proprietary system.
• Analyzed agreement using descriptive statistics and classification tables; assessed impact on vaccine confidence by race/ethnicity.

Main Results:

• The custom system identified 40.60% valid cases; Qualtrics identified 55.21%.
• Agreement between methods was "fair" to "minimal" (κ=0.25).
• Fraud detection method choice influenced vaccine confidence distribution by racial/ethnic group.

Conclusions:

• Fraud detection method selection significantly impacts study sample composition.
• A multilayered approach, combining automated detection with human review tailored to context, is suggested for future surveys.
• Ensuring data integrity is crucial for maximizing the value of web-based survey research.