Mode combinability: Exploring convex combinations of permutation aligned models.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Neural network training often results in multiple models with similar performance but different parameters.
• Linear mode connectivity suggests that models can be interpolated linearly in parameter space while maintaining performance.

Purpose of the Study:

• To explore element-wise convex combinations of neural network parameters.
• To investigate if the concept of linear mode connectivity generalizes to broader phenomena.
• To analyze properties of these model combinations, including transitivity and robustness.

Main Methods:

• Element-wise convex combinations of two permutation-aligned neural network parameter vectors (ΘA and ΘB).
• Extensive experiments with model combinations across the hypercube [0,1]d and its vicinity.
• Analysis of functional and weight similarity of resulting model combinations.

Main Results:

• Broad regions within the hypercube exhibit low loss values, indicating "mode combinability."
• Demonstrated a transitivity property: models re-based to a common third model are also linearly mode connected.
• Showed robustness: perturbed neuron matchings still yield working models.
• Confirmed that model combinations are non-vacuous due to significant functional differences.

Conclusions:

• Mode combinability is a more general phenomenon than linear mode connectivity.
• Neural network parameter spaces possess rich structures allowing for effective model combinations.
• These findings have implications for understanding generalization, model merging, and training dynamics.