Why Empirical Risk Minimization Performs Well for Open Set Domain Adaptation: A Theoretical Analysis From Causal View

Area of Science:

• Machine Learning
• Computer Vision
• Causal Inference

Background:

• Open set domain adaptation (OSDA) addresses unknown classes and distribution shifts.
• Empirical risk minimization (ERM) shows surprising state-of-the-art performance despite theoretical gaps.
• Existing theories fail to fully explain ERM's effectiveness in OSDA.

Purpose of the Study:

• To bridge the theoretical gap in understanding ERM's success in OSDA.
• To develop a causal theoretical framework for OSDA.
• To introduce novel concepts: fully informative causal invariance model (FICIM) and partially informative causal invariance model (PICIM).

Main Methods:

• Formulation of FICIM and PICIM concepts.
• Derivation of a theoretical bound for OSDA.
• Extensive experiments on FICIM and PICIM source domains across diverse datasets.

Main Results:

• ERM performs well when the source domain follows FICIM.
• ERM performs poorly when the source domain follows PICIM.
• Theoretical results are validated by experimental findings.

Conclusions:

• The causal structure of the source domain significantly impacts ERM performance in OSDA.
• The derived theoretical bound explains ERM's varying effectiveness based on information availability.
• Findings offer insights for training and fine-tuning large language models (LLMs).