Robust Subjective Visual Property Prediction from Crowdsourced Pairwise Labels.

Area of Science:

• Computer Vision
• Machine Learning
• Human-Computer Interaction

Background:

• Estimating subjective visual properties (e.g., interestingness) from images/videos is increasingly important.
• Annotation is challenging due to ambiguity; crowdsourcing pairwise comparisons is common but introduces outliers.
• Existing outlier detection methods like majority voting are local and can miss global ranking inconsistencies.

Purpose of the Study:

• To develop a principled method for identifying annotation outliers in subjective visual property estimation.
• To jointly address outlier detection and learning to rank for improved accuracy and efficiency.
• To enable learning with extremely sparse annotations.

Main Methods:

• Formulating subjective visual property prediction as a unified robust learning to rank problem.
• Integrating local pairwise comparison labels to minimize global ranking inconsistency.
• Solving outlier detection and learning to rank simultaneously.

Main Results:

• The proposed method achieves better detection of annotation outliers compared to existing approaches.
• Jointly solving outlier detection and learning to rank leads to improved prediction accuracy.
• The approach is effective even with extremely sparse annotation data.

Conclusions:

• A unified robust learning to rank framework offers a principled approach to subjective visual property estimation.
• Jointly optimizing outlier detection and ranking improves model robustness and data efficiency.
• This method enhances the reliability of subjective visual property prediction from crowdsourced data.