A Multimodal Framework for Understanding Collaborative Design Processes

Area of Science:

• Human-Computer Interaction
• Design Studies
• Computer-Supported Cooperative Work

Background:

• Analyzing collaborative design processes is crucial for understanding outcomes and decision-making.
• Current methods often rely on consolidated textual data from interviews or observations, facing challenges with data volume and heterogeneity.
• Integrating diverse data sources from workshops remains a significant challenge in design research.

Purpose of the Study:

• To propose a practical, modular, and adaptable framework for workshop analysis.
• To introduce an interactive visual analysis system, reCAPit, for multimodal data integration.
• To enhance the methodology for data-rich acquisition, AI-based extraction, and visual analysis of collaborative design workshops.

Main Methods:

• Developed a framework encompassing workshop setup, multimodal data acquisition, AI-based artifact extraction, and visual analysis.
• Created the reCAPit system, integrating modalities like video, audio, notes, and gaze data.
• Conducted six workshops across various themes and analyzed two case studies in detail.

Main Results:

• The reCAPit system enables flexible combination of multimodal data for analysis.
• A multimodal streamgraph visualizes activity and attention, while topic cards summarize discussions.
• Drill-down techniques allow inspection of raw data from various sources.

Conclusions:

• The proposed framework extends collaborative design workshop methodology with data-rich acquisition.
• Combined AI-based extraction and interactive visual analysis offer transparent results dissemination.
• This approach facilitates a deeper understanding of collaboration and decision-making in design processes.