Can Large Language Models Simulate Spoken Human Conversations?

Area of Science:

• Computational linguistics
• Artificial intelligence
• Human-computer interaction

Background:

• Large language models (LLMs) demonstrate advanced capabilities in emulating human cognition, particularly in chat-based interactions.
• The ability of LLMs to simulate spoken human conversation remains largely unexplored, despite their potential as a paradigm shift.

Purpose of the Study:

• To investigate the extent to which large language models (LLMs) can accurately simulate spoken human conversation.
• To compare the linguistic features of LLM-generated conversations with human telephone conversations.

Main Methods:

• Study 1: Compared transcripts from the Switchboard (SB) corpus (human telephone conversations) with transcripts generated by LLMs (GPT-4, Claude Sonnet 3.5, Vicuna, Wayfarer) using tailored prompts.
• Analysis focused on alignment (conceptual, syntactic, lexical), coordination markers, and conversational openings/closings.
• Study 2: Assessed human ability to distinguish LLM-generated transcripts from human SB transcripts.

Main Results:

• LLM conversations exhibited exaggerated alignment, increasing as the conversation progressed, unlike human conversations.
• LLMs demonstrated different and often inappropriate use of coordination markers and dissimilar conversational openings and closings.
• Human evaluators could distinguish LLM-generated transcripts from human conversations, indicating LLMs did not consistently pass for human dialogue.

Conclusions:

• Spoken conversations generated by current LLMs are qualitatively and quantitatively different from human conversations.
• Differences may stem from inherent distinctions between spoken dialogue and chat, or limitations in LLM training and capabilities.
• Future advancements in LLMs and training data may improve simulation, but fundamental differences may persist.