Cross-validation strategies under data dependency: An example with anemia prediction in sheep using ocular...

Area of Science:

• Veterinary Medicine
• Machine Learning
• Computer Vision

Background:

• Computer vision is vital for livestock health monitoring, but its predictive accuracy relies heavily on robust validation methods.
• Ensuring generalizable predictions across different farms and management conditions is essential for real-world applications.

Purpose of the Study:

• To evaluate the impact of three cross-validation (CV) strategies on the performance of a deep learning model for identifying anemic sheep using ocular images.
• To compare random splitting (Rsplit), animal-based splitting (Asplit), and farm-based splitting (Fsplit) for model validation.

Main Methods:

• A dataset of 1176 smartphone images from 186 sheep across three farms was used.
• Images were processed using U-Net for segmentation and VGG19 for anemia prediction based on packed cell volume (PCV).
• Three CV strategies (Rsplit, Asplit, Fsplit) were applied to assess model performance using metrics like accuracy, precision, recall, F1 score, and specificity.

Main Results:

• Random splitting (Rsplit) generally yielded higher performance metrics, potentially overestimating model accuracy.
• Animal-based splitting (Asplit) showed a 24% reduction in accuracy compared to Rsplit.
• Farm-based splitting (Fsplit) resulted in significant drops in accuracy (34%) and F1 score (15%), indicating reduced predictive ability under new farm conditions.

Conclusions:

• The choice of CV strategy significantly influences the estimated performance of computer vision models in livestock farming.
• Farm-level validation provides more conservative and realistic estimates of model generalizability for veterinary applications.
• Aligning CV strategy with the deployment scenario is critical for developing reliable and robust AI tools for animal health.