Comprehension and engagement in survey interviews with virtual agents

Area of Science:

• Human-Computer Interaction
• Survey Methodology
• Artificial Intelligence in Social Science Research

Background:

• Virtual agents are increasingly used in data collection, but their design impacts user interaction.
• Understanding how virtual agent features influence survey respondent comprehension and engagement is crucial for data quality.

Purpose of the Study:

• To investigate the effects of an onscreen virtual agent's dialogue capability and facial animation on survey respondent comprehension and engagement.
• To inform the design of virtual agents for survey interviews, considering different question types.

Main Methods:

• 73 participants were randomly assigned to four interview conditions varying in virtual agent dialogue capability (high/low) and facial animation (high/low).
• Interviews used questions from US government surveys with fictional scenarios to measure comprehension accuracy.
• Participant behavior (eye contact, acknowledgments, smiles) and agent ratings were recorded.

Main Results:

• Higher dialogue capability significantly improved response accuracy and led to more social interaction (e.g., eye contact, perceived personal connection).
• Greater facial animation increased engagement displays (acknowledgments, smiles) but did not improve accuracy and made the agent seem less natural.
• Respondents requested clarification more often with high-dialogue agents, especially for ambiguous scenarios.

Conclusions:

• Virtual agent dialogue capability and facial animation have distinct effects on respondent experience, behavior, and comprehension accuracy.
• Dialogue capability is key for improving survey data accuracy and respondent engagement.
• Design considerations for survey agents should align with the nature of survey questions asked.