Layerwised multimodal knowledge distillation for vision-language pretrained model

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Transformer models excel in multimodal applications by processing both images and text.
• Large model sizes pose challenges for deployment on resource-limited devices, necessitating model compression techniques like knowledge distillation.
• Existing knowledge distillation methods for multimodal models often lead to student model overfitting and struggle with inter-modality interference.

Purpose of the Study:

• To propose a novel layer-wise multimodal knowledge distillation method for vision-language pretrained models.
• To address the limitations of existing knowledge distillation techniques, specifically overfitting and modality interference.
• To improve the efficiency and performance of multimodal models for deployment on resource-constrained devices.

Main Methods:

• Implemented layer-wise knowledge distillation, utilizing both intermediate and final layers of the teacher model.
• Separated modalities and provided them as extra inputs to mitigate mutual interference.
• Introduced two auxiliary losses to enhance the effectiveness of distillation for each modality.

Main Results:

• The proposed layer-wise multimodal knowledge distillation significantly outperformed existing knowledge distillation methods.
• Demonstrated superior performance across four diverse multimodal tasks.
• Successfully transferred knowledge from large teacher models to smaller student models without performance degradation.

Conclusions:

• Layer-wise multimodal knowledge distillation is an effective strategy for compressing large vision-language pretrained models.
• The proposed method mitigates overfitting and modality interference, leading to improved student model performance.
• This approach facilitates the deployment of powerful multimodal AI on devices with limited computational resources.