A Stationary (And Therefore Compatible) Representation is All You Need

Area of Science:

• Machine Learning
• Computer Vision
• Artificial Intelligence

Background:

• Learning compatible representations is crucial for models that are updated over time.
• Sequential fine-tuning poses challenges for maintaining representation compatibility.
• Existing methods may not capture higher-order dependencies between model updates.

Purpose of the Study:

• To demonstrate that stationary representations from d-Simplex fixed classifiers imply compatibility.
• To develop methods for learning compatible representations during sequential model fine-tuning.
• To improve the robustness and performance of models undergoing updates and replacements.

Main Methods:

• Utilizing d-Simplex fixed classifiers for learning stationary representations.
• Employing a convex combination of cross-entropy and contrastive loss.
• Conducting extensive experiments on sequential fine-tuning and model replacement scenarios.

Main Results:

• Stationary representations learned by d-Simplex fixed classifiers satisfy the formal definition of compatibility.
• The proposed loss combination captures higher-order dependencies, ensuring compatibility.
• Achieved state-of-the-art performance in scenarios with sequential model updates and replacements.

Conclusions:

• Stationary representations are foundational for compatible representations in machine learning.
• The combined loss function effectively addresses challenges in sequential model updates.
• Compatible representations enable uninterrupted services and improved performance during model evolution.