Storing, combining and analysing turkey experimental data in the Big Data era

Area of Science:

• Livestock data management
• Animal science
• Data engineering

Background:

• Increasing data volumes in livestock require efficient storage and analysis solutions.
• Traditional methods struggle with scalability and interoperability.
• Automating animal gait scoring presents a data-intensive challenge.

Purpose of the Study:

• To explore the utility of a data lake for livestock data.
• To assess the scalability of data preprocessing (ETL) procedures.
• To develop a machine learning pipeline for automated gait score classification.

Main Methods:

• Deployed a data lake using inertial measurement unit (IMU) and force plate (FP) data.
• Implemented an extract, transform, load (ETL) procedure for data preprocessing.
• Simulated increasing data volumes to test ETL scalability and developed a machine learning pipeline.

Main Results:

• The ETL procedure demonstrated scalability, with processing time reduced significantly using multiple cores.
• A machine learning pipeline was successfully developed to classify gait scores.
• The data lake effectively handled and analyzed diverse data types.

Conclusions:

• A data lake is a viable solution for managing large, diverse livestock datasets.
• The implemented ETL pipeline is scalable and efficient.
• Data lakes facilitate the development of machine learning models for animal welfare applications.