Providing a Single Ground-Truth for Illuminant Estimation for the ColorChecker Dataset

Area of Science:

• Computer Vision
• Image Processing
• Color Science

Background:

• The ColorChecker dataset is a standard benchmark for evaluating illuminant estimation algorithms.
• Existing ground-truth data for the ColorChecker dataset is inconsistent, with at least three different sets available.
• This inconsistency can lead to conflicting conclusions about algorithm performance in scientific literature.

Purpose of the Study:

• To investigate the reasons behind the discrepancies in the existing ground-truth data for the ColorChecker dataset.
• To introduce a new, unified, and recommended set of ground-truth data for the ColorChecker dataset.
• To demonstrate the impact of different ground-truth datasets on the performance evaluation of illuminant estimation algorithms.

Main Methods:

• Analysis of the sources of variation among the three existing ground-truth datasets.
• Development and validation of a new, single, and recommended ground-truth dataset.
• Comparative experiments evaluating illuminant estimation algorithms using both legacy and the new ground-truth data.

Main Results:

• The study identifies the causes for the differences in the existing ground-truth datasets.
• A new, recommended ground-truth dataset is proposed.
• Experiments show that the ranking of illuminant estimation algorithms can be reversed based on the ground-truth data used.

Conclusions:

• The choice of ground-truth data significantly impacts the evaluation and ranking of illuminant estimation algorithms.
• A standardized and validated ground-truth dataset is essential for reliable benchmarking.
• The proposed unified ground-truth dataset provides a more consistent and accurate basis for algorithm development and comparison.