Deep Model Fusion: A Survey.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Deep model fusion integrates multiple deep learning (DL) models to improve performance by combining their strengths and mitigating individual model weaknesses.
• Challenges exist in deep model fusion, particularly for large-scale models like large language models (LLMs) and foundation models, due to high computational costs and model heterogeneity.

Purpose of the Study:

• To provide a comprehensive survey of recent advancements in deep model fusion techniques.
• To categorize and analyze existing model fusion methods and identify future research directions.

Main Methods:

• Categorization of deep model fusion methods into four main types: weight averaging (WA), mode connectivity, alignment, and ensemble learning.
• Weight averaging: Merging model parameters by averaging them to approximate optimal solutions.
• Mode connectivity: Transforming models along paths of non-increasing loss to achieve consistent functions and better fusion.
• Alignment: Matching corresponding units between models before merging to leverage inter-model relationships.
• Ensemble learning: Combining predictions from multiple models during inference to boost accuracy and robustness.

Main Results:

• Existing deep model fusion techniques can be broadly classified into parameter-level fusion (WA, mode connectivity, alignment) and inference-level fusion (ensemble learning).
• Each method offers distinct advantages for integrating diverse deep learning models.

Conclusions:

• Deep model fusion is a promising technique for enhancing DL model performance, especially for large-scale architectures.
• Addressing challenges like computational cost and model interference is crucial for advancing the field.
• Future research should explore novel fusion strategies and optimization techniques for heterogeneous and large-scale models.