A Deixis-Centered Approach for Documenting Remote Synchronous Communication Around Data Visualizations

Area of Science:

• Human-Computer Interaction
• Data Visualization
• Linguistics

Background:

• Referential gestures, or deixis, are crucial in data visualization communication but often omitted in meeting documentation.
• Current documentation methods like transcripts and videos inadequately capture or emphasize these gestural elements.

Purpose of the Study:

• To introduce a novel framework for documenting collaborative data meetings that integrates deixis as a primary component.
• To develop an automated system for capturing and annotating gestural data within interactive documents.

Main Methods:

• Capturing cursor-based gestural data alongside audio recordings.
• Utilizing a large language model to correlate spoken words with identified gestures.
• Converting captured data into interactive documents with context-based annotations.

Main Results:

• Participants showed a strong preference for the automated interactive documentation over traditional methods (recordings, transcripts, manual notes).
• A preliminary taxonomy of cursor-based deictic gestures was derived from user study observations.

Conclusions:

• The proposed framework effectively documents collaborative data meetings by treating deixis as a first-class citizen.
• The developed interactive documentation and gesture taxonomy offer advancements for collaborative data analysis.