Leveraging ubiquitous mobile sensors to track infant expressions and movements

Capturing the complexity of infant behavior in naturalistic contexts remains a primary challenge in developmental science. High-density sensing can address this, but its adoption has been limited by the need for specialized, expensive, and physically intrusive hardware. We developed Expressions, an iOS application and data processing pipeline that repurposes Apple's TrueDepth camera system as a contactless, multimodal data collection tool. The application passively collects 60 Hz data on over 50 low-level features of facial expressions, 3D head movements for up to three interacting individuals, and eye gaze direction, while concurrently recording synchronized audio and video of the environment. We deployed Expressions in two developmental studies to test the feasibility and flexibility of this approach. In the first study, toddlers watched emotional videos on a computer that wirelessly streamed event markers to the recording device. This timestamping allowed us to time-lock and analyze the toddlers' micro-expressive and emotional responses to positive versus negative stimuli. In the second study, two wirelessly networked iPads captured simultaneous data from caregiver-infant dyads during free-flowing play. This markerless, dual-device setup allowed assessment of complex interaction dynamics without disrupting the interaction itself. These studies show that consumer mobile devices are a feasible and flexible platform for studying infant behavior. Because the facial tracking in Expressions runs entirely on-device, the app does not require internet access during data collection. This privacy feature allows deployment not only in labs and urban homes but also in remote and rural communities that have been underrepresented in developmental research.