OoDBench+: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalization.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Deep learning excels with independent and identically distributed (i.i.d.) data.
• Significant challenges arise with Out-of-Distribution (OoD) generalization, where data distributions differ.
• Current evaluation methods for OoD generalization algorithms are limited.

Purpose of the Study:

• To identify and formally define two key types of distribution shifts: diversity shift and correlation shift.
• To analyze how these shifts impact the performance of Out-of-Distribution generalization algorithms.
• To provide a unified framework for evaluating OoD generalization across various datasets and tasks.

Main Methods:

• Formal definition of diversity shift and correlation shift.
• Empirical evaluation of existing OoD generalization algorithms on datasets dominated by each shift type.
• Analysis of performance degradation attributed to the defined shifts.

Main Results:

• Diversity and correlation shifts are ubiquitous in OoD datasets and upper bound algorithm performance.
• Existing OoD algorithms show varying strengths and limitations against each shift type.
• All performance degradations in OoD settings can be explained by these two defined shifts.

Conclusions:

• The proposed diversity and correlation shifts provide a fundamental understanding of Out-of-Distribution generalization challenges.
• This work establishes a benchmark for evaluating and developing more robust OoD generalization algorithms.
• The findings pave the way for future research in reliable deep learning across different data distributions.