Identifying and Analyzing Bot-Generated Responses in Online Health Care Surveys: Methodological Study.

Area of Science:

• Health Services Research
• Digital Health
• Data Integrity

Background:

• Increasing reliance on online surveys for patient-reported outcomes in health care research.
• Growing concerns regarding fraudulent bot responses threatening data integrity and validity.
• Potential for distorted statistical analyses and misinformed health policy decisions.

Purpose of the Study:

• Develop criteria to identify bot-generated responses in online health care surveys.
• Examine the impact of bot responses on data quality and survey results.
• Compare survey data between probable human and suspected bot respondents.

Main Methods:

• Conducted an online survey (July-October 2023) on health care use, patient experiences, and outcomes.
• Developed a 3-tier classification system using 'red flags' (e.g., duplicate responses, timestamp inconsistencies, location discrepancies) to detect bots.
• Utilized chi-square and Spearman correlation tests for quantitative analysis of differences and relationships.

Main Results:

• 58% of 1154 responses were classified as suspected bot-generated.
• Duplicate open-ended responses were the most frequent bot indicator (44%).
• Significant differences observed between bots and humans; bots favored middle Likert scale responses, while humans chose extremes. Expected relationships between health indicators were reversed in bot data.

Conclusions:

• Implementing bot prevention and detection is crucial for preserving data integrity in online health surveys.
• Failure to detect bots risks distorting research findings, especially in health equity studies.
• Effective bot detection strategies include open-text analysis, timestamp evaluation, and geographic validation; ongoing advancements are necessary.