Dense Paraphrasing for multimodal dialogue interpretation.

Area of Science:

• Computational Linguistics
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Multimodal dialogue presents computational challenges due to complex interactions between speech, gesture, action, and gaze.
• Traditional dialogue systems struggle to accurately track and interpret these intertwined modalities.
• Existing methods often require specialized multimodal models.

Purpose of the Study:

• To extend Dense Paraphrasing (DP) by translating nonverbal modalities into linguistic expressions.
• To simplify multimodal information representation for enhanced computational understanding of situated dialogues.
• To evaluate the effectiveness of DP for the Common Ground Tracking (CGT) problem using instruction-tuned Large Language Models (LLMs).

Main Methods:

• Translated nonverbal dialogue modalities (gesture, gaze, action) into linguistic expressions.
• Utilized Dense Paraphrasing (DP) to create a compact, machine-readable text format for multimodal dialogue.
• Evaluated language-only LLMs on the Common Ground Tracking (CGT) task using a collaborative problem-solving dataset.

Main Results:

• Dense paraphrased language form effectively improved LLM performance on the CGT task.
• The DP technique enabled language-only models to process and integrate multimodal information.
• Augmenting context with DP significantly improved common ground reasoning compared to baselines.

Conclusions:

• Dense Paraphrasing (DP) offers a robust method for interpreting and integrating multimodal communication subtleties.
• This approach enhances dialogue system performance by enabling better understanding of situated dialogues.
• DP facilitates effective common ground reasoning, paving the way for improved real-world dialogue systems.