Aligning Video Models with Human Social Judgments via Behavior-Guided Fine-Tuning

Area of Science:

• Artificial Intelligence
• Cognitive Science
• Computer Vision

Background:

• Humans intuitively process complex social signals in visual contexts.
• Current AI models' ability to encode human-like social similarity is not well understood.
• A gap exists between AI's visual processing and human social perception.

Purpose of the Study:

• To investigate if AI models capture human-perceived similarity in social videos.
• To develop methods for instilling human social similarity structures into AI models using behavioral data.
• To address the modality gap where language models outperform video models in social similarity tasks.

Main Methods:

• Created a benchmark of over 49,000 human similarity judgments on social interaction videos.
• Introduced a novel hybrid triplet-RSA objective with low-rank adaptation (LoRA) for fine-tuning.
• Fine-tuned a TimeSformer video model using human judgments to align pairwise distances with human similarity.

Main Results:

• Discovered a modality gap: language model embeddings better matched human similarity than video models.
• Fine-tuned video models showed significantly improved alignment with human perceptions on held-out data.
• Fine-tuning enhanced the encoding of social-affective attributes like intimacy, valence, dominance, and communication.

Conclusions:

• Pretrained video models exhibit a deficit in social recognition capabilities.
• Behavior-guided fine-tuning effectively shapes video representations to align with human social perception.
• This approach bridges the gap between AI and human understanding of social dynamics in videos.