An Approach to Aligning Categorical and Continuous Time Series for Studying the Dynamics of Complex Human Behavior

An emerging perspective on human cognition and performance sees it as a kind of self-organizing phenomenon involving dynamic coordination across the body, brain and environment. Measuring this coordination faces a major challenge. Time series obtained from such cognitive, behavioral, and physiological coordination are often complicated in terms of non-stationarity and non-linearity, and in terms of continuous vs. categorical scales. Researchers have proposed several analytical tools and frameworks. One method designed to overcome these complexities is recurrence quantification analysis, developed in the study of non-linear dynamics. It has been applied in various domains, including linguistic (categorical) data or motion (continuous) data. However, most previous studies have applied recurrence methods individually to categorical or continuous data. To understand how complex coordination works, an integration of these types of behavior is needed. We aimed to integrate these methods to investigate the relationship between language (categorical) and motion (continuous) directly. To do so, we added temporal information (a time stamp) to categorical data (i.e., language), and applied joint recurrence analysis methods to visualize and quantify speech-motion coordination coupling during a rap performance. We illustrate how new dynamic methods may capture this coordination in a small case-study design on this expert rap performance. We describe a case study suggesting this kind of dynamic analysis holds promise, and end by discussing the theoretical implications of studying complex performances of this kind as a dynamic, coordinated phenomenon.