Learning Invariant Representations using Inverse Contrastive Loss.

Area of Science:

• Machine Learning
• Representation Learning

Background:

• Learning invariant representations is crucial for machine learning.
• Current methods like the information bottleneck principle struggle with metric structure optimization.

Purpose of the Study:

• Introduce a novel loss function, inverse contrastive loss (ICL), for learning invariant representations.
• Address limitations of existing methods by incorporating metric structure.

Main Methods:

• Invert contrastive losses to formulate ICL.
• Demonstrate equivalence to regularized MMD divergence for binary extraneous variables.
• Show ICL decomposes into convex functions of distance metrics for general cases.

Main Results:

• Models optimized with ICL exhibit superior invariance to extraneous variables.
• Maintained desired accuracy levels while improving invariance.
• Successfully applied ICL to both continuous and discrete extraneous variables.

Conclusions:

• ICL provides an effective and optimizable approach for learning invariant representations.
• The method is versatile and applicable across various data types and variable types.