Multi-Modal Feature Fusion and Hierarchical Classification for Automated Equine-Human Interaction Behavior...

Area of Science:

• Computational ethology and computer vision for animal behavior analysis.
• Human-animal interaction (HAI) research focusing on equine-human dynamics.

Background:

• Automated recognition of equine-human interaction is challenging due to data imbalance and the need for cross-species analysis.
• Existing methods often ignore full-body posture or fail with imbalanced datasets, lacking integrated human and equine body language analysis.

Purpose of the Study:

• To develop a novel hierarchical classification framework for distinguishing behavioral states in horse-human encounters.
• To integrate multi-modal computer vision features for simultaneous analysis of human and equine body language.

Main Methods:

• Utilized a hierarchical classification framework with three feature extraction pipelines: YOLOv8 (spatial relationships), MediaPipe (human pose), and AP-10K (equine pose).
• Extracted 35 discriminative features from 28 videos (50,270 samples) across five horse breeds.
• Employed cost-sensitive gradient boosting with automatic class weight optimization to address severe class imbalance (18.3:1 ratio).

Main Results:

• The first stage achieved 73.2% balanced accuracy classifying interactions into affiliative, neutral, and avoidant categories.
• The second stage achieved 88.5% balanced accuracy for six fine-grained sub-behaviors (with oracle routing).
• Achieved 85.0% recall for safety-critical avoidant behaviors (3.8% of data), demonstrating effectiveness on imbalanced classes.

Conclusions:

• The proposed framework is the first systematic multimodal cross-species behavioral assessment pipeline for human-animal interaction.
• Equine pose features were critical for classification performance, highlighting the importance of species-specific indicators.
• The system has direct implications for enhancing equine welfare monitoring and rider safety protocols.