Automated survey collection with LLM-based conversational agents

Phone surveys are crucial for collecting health data but are expensive, time-consuming, and difficult to scale. To overcome these limitations, we propose a survey collection approach powered by conversational Large Language Models (LLMs).

Our framework leverages an LLM-powered conversational agent to conduct surveys and transcribe conversations, along with an LLM (GPT-4o) to extract responses from the transcripts. We evaluated the framework's performance by analyzing transcription errors, the accuracy of inferred survey responses, and participant experiences across 40 survey responses collected from a convenience sample of 8 individuals, each adopting the role of five LLM-generated personas.

GPT-4o extracted responses to survey questions with an average accuracy of 98%, despite an average transcription word error rate of 7.7%. Participants reported occasional errors by the conversational agent but praised its ability to demonstrate comprehension and maintain engaging conversations.

Our study showcases the potential of LLM agents to enable scalable, AI-powered phone surveys, reducing human effort and advancing healthcare data collection.