Cattle lameness detection using depth image and deep learning

Lameness in cattle is a significant welfare and economic concern. To address this, we developed an end-to-end deep learning framework for 24/7 lameness monitoring using top-down depth images of cattle. The framework integrates three key stages: instance segmentation for detection, a custom multi-object tracking algorithm for identity preservation, and a spatio-temporal model for classification. We compared multiple instance segmentation models (Mask R-CNN, YOLOv8m-seg, YOLOv11m-seg) and evaluated three proposed tracking algorithms version1, 2 and 3 (PTAV1, PTAV2, and PTAV3). For classification, we tested multiple configurations integrating various pre-processing conditions (no filter, Gaussian, median), seven EfficientNet backbones (B1-B7), two temporal sequence lengths (5 and 7 frames), and a Long Short-Term Memory (LSTM) network to assign a lameness score from 1 (healthy) to 4 (lame) based on expert ground truth. In the detection model comparison, the YOLOv11m-seg model emerged as the top performer for detection, achieving a BBox AP@50 of 99.38%, Mask AP@50 of 99.26%, at 75.49 FPS. Our proposed tracking algorithm, PTAV3, which leverages location and direction prediction, achieved an exceptional overall accuracy of 99.94% (95% CI: 99.7-100%). For classification, the best model-an EfficientNet-B7 + LSTM architecture-yielded an accuracy of 95.95% (95% CI: 94.8-97.1%) and an F1-score of 96.06% (95% CI: 94.8-97.1%) on unseen test data, using a 5-frame sequence with no pre-processing filter. This integrated system provides a robust, automated, and objective solution for lameness scoring, showcasing the potential for real-time animal welfare monitoring in agricultural settings.