What Do Different Evaluation Metrics Tell Us About Saliency Models?

Area of Science:

• Computer Vision
• Human-Computer Interaction
• Cognitive Science

Background:

• Accurately predicting human visual attention in images is crucial for AI development.
• Current methods for evaluating saliency models lack standardized metrics, leading to inconsistent results.

Purpose of the Study:

• To analyze and compare the properties of eight different evaluation metrics for image saliency models.
• To provide a transparent and interpretable framework for assessing saliency prediction accuracy.

Main Methods:

• Systematic experimentation with eight distinct saliency evaluation metrics.
• Visualization techniques to illustrate metric computations and their impact on saliency scores.
• Comparative analysis of metric behavior based on factors like false positives/negatives and spatial deviations.

Main Results:

• Significant differences in how metrics rank saliency models were observed.
• Metric performance is influenced by pre-processing, handling of viewing biases, and spatial accuracy.
• The study visualizes metric computations, enhancing the interpretability of saliency scores.

Conclusions:

• The choice of evaluation metric critically impacts saliency model assessment.
• Recommendations are provided for selecting appropriate metrics based on specific application needs and assumptions.
• Increased transparency in saliency model evaluation is facilitated by understanding metric properties.