The MultiSOCIAL Toolbox: An open-source toolkit for advancing multimodal interaction research

Human interaction is intrinsically multimodal, in that it unfolds across different kinds of behaviors (or modalities). However, many researchers have found it difficult to systematically investigate the richness of multimodal communication due to its theoretical and technical complexity. To help overcome some of the technical hurdles, we introduce an open-source single-platform solution: the MultiSOCIAL (Multimodal timeSeries Open-SourCe Interaction Analysis Library) Toolbox. The Toolbox enables any researcher who has video files to extract time-series data in three modalities: body movement to quantify non-verbal behavior through a pose estimation algorithm; transcripts of what was said during an interaction through the use of an automatic speech recognition system; and acoustic prosodic characteristics of speech through the use of audio signal processing. The toolkit uses existing gold-standard open-source tools, and we have brought them together in a freely available graphical user interface (GUI) that requires no coding and runs exclusively on the user's machine. To provide a concrete use-case, we present an analysis of a new dataset collected by undergraduate students as part of a seminar class. Here, we specifically analyze the movement data from the study to exemplify the complexity available in video data from just one modality. We find that friends show higher interpersonal bodily coordination than strangers and that this coordination is both more stable and more complex throughout a simple affiliative conversation. We end with some practical recommendations for using the Toolbox, including how to best set up experiments to get the most accurate data possible to maximize data quality.