Crowdtruth validation: a new paradigm for validating algorithms that rely on image correspondences.

Area of Science:

• Medical Image Computing
• Computer-Assisted Interventions
• Surgical Technology

Background:

• Feature tracking and 3D surface reconstruction are crucial for computer-assisted minimally invasive surgery.
• A significant challenge in developing these technologies is the scarcity of large, annotated image datasets reflecting real-world clinical variations.

Purpose of the Study:

• To develop a novel, cost-effective, and rapid method for generating high-quality reference image annotations.
• To overcome the bottleneck of limited annotated data for training and validating surgical algorithms.

Main Methods:

• A four-stage crowdsourcing approach was employed, involving feature detection, correspondence search by online users, merging annotations using Gaussian finite mixture models, and outlier removal.
• The method leverages an online community for annotation tasks.

Main Results:

• Approximately 10,000 annotations were generated within 24 hours at a cost of $100.
• Crowdsourced annotations, after processing, achieved expert quality with a median error of approximately 1 pixel compared to reference data.
• The outlier removal threshold influences annotation accuracy and data selection.

Conclusions:

• The proposed crowdsourcing concept offers a novel, efficient, and accurate solution for generating correspondence data.
• This method is suitable for large-scale data annotation in medical image computing and computer-assisted interventions.
• The approach enables fast, low-cost, and highly accurate annotation generation.