Can intelligent agents improve data quality in online questiosnnaires? A pilot study

Area of Science:

• Human-Computer Interaction
• Survey Methodology
• Data Quality Assurance

Background:

• Online surveys are a common data collection method.
• Ensuring high data quality in online surveys can be challenging.
• Chatbots offer a potential solution for improving user experience and data accuracy.

Purpose of the Study:

• To investigate the effectiveness of chatbots in enhancing data quality for online surveys.
• To assess participant utilization and satisfaction with chatbot assistance.
• To determine if chatbot support influences the accuracy of responses to challenging survey items.

Main Methods:

• A randomized controlled trial with 300 Australian adults recruited via Prolific Academic.
• Comparison between a chatbot-supported online questionnaire group and an unassisted control group.
• Inclusion of validated measures and specifically designed confusing challenge items.
• Measurement of chatbot usage, user satisfaction, and data quality through session logs, feedback, and analysis of response relationships.

Main Results:

• The majority of participants with chatbot access utilized the feature.
• Most users reported finding the chatbot helpful and expressed satisfaction.
• A modest improvement in data quality was observed in the chatbot-supported group compared to the control group.
• The effect size may have been underestimated due to specific item characteristics.

Conclusions:

• Assistive chatbots are a valuable tool for improving data quality in online surveys.
• Chatbot features are likely to be utilized by a significant portion of participants when available.
• User perception of chatbots in survey contexts is generally positive, indicating their potential for broader adoption.
• Further research with more complex questionnaires is recommended to fully understand chatbot utility.