Conversational presentation mode increases credibility judgements during information search with ChatGPT

Area of Science:

• Human-Computer Interaction
• Information Science
• Cognitive Psychology

Background:

• Conversational agents powered by large language models (LLMs) are increasingly used for information retrieval.
• Ensuring users can accurately assess the credibility of LLM-generated information is crucial due to potential factual inaccuracies.
• Understanding factors influencing user credibility judgments is essential for mitigating misinformation.

Purpose of the Study:

• To investigate how the presentation mode of information (conversational agent vs. static text) affects users' ability to detect inaccuracies.
• To compare credibility judgments across different types of conversational agents (text-based, voice-based) and a static online encyclopedia.
• To identify whether the conversational nature of LLM agents hinders accurate credibility assessment.

Main Methods:

• Conducted two preregistered experiments involving participants rating the credibility of accurate and partially inaccurate information.
• Information was presented via a dynamic text-based LLM agent, a voice-based agent, or a static text-based online encyclopedia.
• Analyzed data using mediation analysis to understand the role of conversational nature in credibility judgments.

Main Results:

• Participants were significantly better at detecting inaccuracies when information was presented as static text compared to both text-based and voice-based conversational agents.
• This effect persisted regardless of whether the information sources were branded (e.g., ChatGPT, Alexa, Wikipedia) or unbranded.
• Mediation analysis supported the hypothesis that the conversational format of LLM agents poses a threat to accurate credibility assessment.

Conclusions:

• The presentation mode of information critically influences users' ability to discern factual accuracy.
• Conversational interfaces, despite their dynamic nature, may inadvertently reduce users' critical evaluation of information.
• Future research and design should consider the impact of presentation modality on combating misinformation effectively.