Crowdsourcing image analysis for plant phenomics to generate ground truth data for machine learning.

Area of Science:

• Agricultural Science
• Computer Science
• Data Science

Background:

• High-quality ground truth data is essential for accurate machine learning (ML) model training.
• Professional data annotation is often expensive and time-consuming.
• Crowdsourcing presents a potential solution for scalable and affordable data generation.

Purpose of the Study:

• To evaluate the effectiveness of crowdsourcing for generating ground truth data for an image analysis task.
• To compare the performance of different crowdsourcing groups (students, Amazon MTurk, Master MTurk) in terms of accuracy and speed.
• To assess the quality of crowdsourced data against expert annotations.

Main Methods:

• An image analysis task involving the segmentation of corn tassels was designed.
• Data was collected from students for academic credit, Amazon Mechanical Turk (MTurk) workers, and Master MTurk workers.
• Accuracy, speed, and other quality metrics were analyzed and compared across groups.

Main Results:

• Amazon MTurk and Master MTurk workers significantly outperformed students in segmentation accuracy.
• No significant difference in performance was observed between Amazon MTurk and Master MTurk workers.
• The quality of segmentation by MTurk workers was comparable to that of an expert annotator.

Conclusions:

• Properly managed crowdsourcing can generate large volumes of high-quality ground truth data cost-effectively.
• Crowdsourcing is particularly viable for applications like high-throughput plant phenotyping.
• Best practices for assessing and comparing data quality from different sources were provided.