Detecting the corruption of online questionnaires by artificial intelligence.

Area of Science:

• Social Sciences
• Computer Science
• Data Science

Background:

• Online questionnaires leverage crowdsourcing for efficiency and cost-effectiveness.
• Advancements in artificial intelligence (AI), specifically large language models (LLMs), enable automated form completion and text generation.
• This poses a significant threat to the integrity of data collected through online surveys.

Purpose of the Study:

• To evaluate the detectability of AI-generated text in online studies by both human evaluators and automated AI detection systems.
• To assess the effectiveness of current methods in ensuring data quality in the face of AI-driven fraudulent submissions.

Main Methods:

• Human participants were tasked with distinguishing between human-written and AI-generated text in an online study context.
• Automated AI detection systems were evaluated for their ability to identify AI-generated responses.
• Accuracy rates for both human and automated detection were recorded and analyzed.

Main Results:

• Human evaluators achieved an accuracy rate of 76% in identifying AI-generated text, which is above chance but insufficient for ensuring high data quality.
• Current automatic AI detection systems proved to be entirely ineffective in detecting AI-generated submissions.
• The study highlights a critical gap in current methodologies for maintaining data integrity in online research.

Conclusions:

• Relying on human attention checks alone is becoming insufficient to guarantee data quality due to the sophistication of AI-generated content.
• Crowdsourcing platforms must develop systematic solutions to address AI-driven data fabrication, as current detection tools are inadequate.
• The increasing prevalence of AI submissions risks making the costs of fraud detection prohibitive, undermining the utility of online questionnaires.