Chatbot Voting Advice Applications inform but seldom sway young unaligned voters

Area of Science:

• Political Science
• Computational Social Science
• Human-Computer Interaction

Background:

• Voting Advice Applications (VAAs) are understudied for their effectiveness in enhancing political knowledge and participation.
• Traditional VAAs may not reach less engaged electorates due to user self-selection.
• Large Language Models (LLMs) offer new possibilities for personalized political information delivery.

Purpose of the Study:

• To introduce and evaluate a novel VAA Bot utilizing LLMs and retrieval-augmented generation.
• To assess the VAA Bot's impact on political knowledge and participation among young, politically unaffiliated adults.
• To understand the limitations of LLM-based tools in influencing vote preferences and party evaluations.

Main Methods:

• Development of a VAA Bot integrating LLMs and retrieval-augmented generation.
• Conducted three experimental studies with young, politically unaffiliated adults.
• Collected data on user knowledge, issue stances, vote preferences, and party evaluations.

Main Results:

• The VAA Bot significantly improved users' knowledge of party stances on personally important issues.
• Weak effects were observed on vote preferences and party evaluations, especially for users with strong pre-existing issue alignments.
• Personalized information delivery enhanced issue knowledge but did not substantially shift political attitudes in all cases.

Conclusions:

• LLM-based VAAs can enhance political knowledge, particularly on specific issues relevant to the user.
• The impact of LLM-powered political tools on downstream behaviors like voting is nuanced and may be limited by existing ideological commitments.
• Future research should explore strategies to broaden the reach and impact of digital civic learning tools.