A modular machine learning tool for holistic and fine-grained behavioral analysis.

Area of Science:

• Computer Vision and Machine Learning
• Behavioral Science
• Human-Computer Interaction

Background:

• Existing AI tools lack holistic, fine-grained human behavior analysis from videos.
• There is a need for accessible tools to analyze complex behavioral data.

Purpose of the Study:

• To develop and validate a novel machine learning tool for comprehensive behavioral analysis from video data.
• To assess the tool's ability to differentiate behaviors across various environmental conditions and emotional stimuli.

Main Methods:

• A two-level machine learning approach combining computer vision (OpenPose, OpenFace) for feature extraction and algorithms (XGBoost, LSTM) for classification.
• Filming 16 participants across six conditions varying presence of people, sound, and emotional stimuli (self-referential vs. control).
• Utilizing explainability to identify key behavioral features driving classification outcomes.

Main Results:

• High classification rates (AUC=0.8-0.9) for person presence vs. sound/silence, with action units and gaze identified as key features.
• Moderate classification rates (AUC=0.7-0.8) for sound vs. silence and self-referential vs. control conditions, linked to specific facial and body point data.
• Findings align with traditional hypothesis-driven approaches, validating the tool's effectiveness.

Conclusions:

• The developed AI tool provides effective holistic and fine-grained behavioral analysis from videos.
• The tool's modularity allows for extension to more complex, naturalistic behavioral research settings.
• This approach offers a promising avenue for advancing behavioral science research through accessible AI.