A Multimodal Dual-Stream Framework for Sheep Behavior Recognition Using Skeletal and Local Visual Fusion

Intelligent sheep behavior monitoring is vital for modern husbandry, but faces severe challenges in natural pastures due to high-density flock occlusion. Traditional 2D skeleton-based networks often suffer from depth ambiguity and feature collapse, misclassifying static tremors as dynamic displacement. To overcome this, we propose a robust multimodal dual-stream framework using skeletal and local visual fusion. The architecture features an upstream spatial perception stage utilizing YOLOv11m-Pose. To reduce annotation costs and improve robustness, we introduce an Active Hard-Example Mining mechanism, explicitly retaining difficult samples with severe overlapping or edge truncation. For downstream behavioral decisions, a multimodal dual-stream architecture processes the targets. The Spatio-Temporal Kinematic Stream employs a Kinematic Denoising Engine, incorporating a 1D Gaussian filter and displacement dead-zone gate to purify 2D coordinates before feeding them into a BiLSTM network. Concurrently, the Spatial Visual Stream uses a ResNet-50 backbone on cropped RGB patches to capture essential spatial context, addressing the limitations of pure coordinates. Finally, a weighted Softmax layer integrates both streams. Experiments on a complex real-world dataset validate this approach. A baseline kinematic-only model achieved just 69.05% overall accuracy and 68.18% walking precision. In contrast, our dual-stream fusion network achieved 93.26% overall accuracy, elevating walking precision to 97.14% and the eating F1-score to 94.29%. By effectively decoupling similar static and dynamic behaviors, this study demonstrates the indispensability of local visual features, establishing a high-precision baseline for smart livestock monitoring.