How reliable is visual lameness scoring? Assessing human label variability for use in automated detection systems.

Area of Science:

• Animal Science
• Veterinary Medicine
• Machine Learning

Background:

• Visual mobility scoring is vital for detecting lameness in dairy cattle.
• Current subjective scoring methods are inconsistent, affecting data quality for machine learning.
• There is a need for reliable methods to assess and label cattle mobility.

Purpose of the Study:

• To assess the reliability of visual mobility scores from multiple assessors.
• To evaluate the influence of scoring methods (live vs. video) and experience levels on label quality.
• To identify factors contributing to scoring variation for machine learning applications.

Main Methods:

• Data collected from two farms using the AHDB 4-point mobility scale and a dichotomized version.
• Both live and video assessments were conducted by assessors with varying experience.
• Statistical analysis, including weighted kappa and regression, was used to assess agreement and influencing factors.

Main Results:

• Substantial within- and between-assessor variation was observed, especially for normal and slightly abnormal gaits.
• Score consistency was only fair (weighted kappa ≈ 0.33) between live and video assessments.
• Increased video viewing frequency and animal speed negatively impacted inter-assessor agreement.

Conclusions:

• Subjective visual mobility scoring presents significant challenges for creating reliable labels for machine learning.
• Simplifying scores improves agreement but reduces data granularity.
• Strategies to minimize scoring variation and enhance label definitiveness are essential for improving automatic lameness detection in dairy cattle.