Towards a better evaluation of out-of-domain generalization

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Domain Generalization (DG) aims to create algorithms that perform well on unseen data distributions.
• The average measure is commonly used to compare DG algorithms but its accuracy is questionable.
• Existing DG studies lack a thorough investigation into the limitations of the average measure.

Purpose of the Study:

• To investigate the limitations of the average measure in Domain Generalization.
• To propose and validate the worst+gap measure as a superior alternative.
• To provide a more reliable method for assessing DG algorithm performance.

Main Methods:

• Theoretical analysis to establish the foundation of the worst+gap measure, including deriving two theorems.
• Development of modified datasets (SR-CMNIST, C-Cats&Dogs, L-CIFAR10, PACS-corrupted, VLCS-corrupted) to accurately evaluate DG performance.
• Extensive experimental comparisons between the proposed worst+gap measure and the conventional average measure.

Main Results:

• The average measure demonstrates inferior performance in approximating true Domain Generalization capabilities.
• The proposed worst+gap measure is shown to be a robust and theoretically supported alternative.
• Experimental results validate the effectiveness of the worst+gap measure over the average measure.

Conclusions:

• The average measure is inadequate for accurately evaluating Domain Generalization performance.
• The worst+gap measure offers a more reliable and theoretically sound approach for DG research.
• This study provides a critical advancement in the evaluation methodologies for Domain Generalization.