DiffuMSMO: A multi-modal diffusion model for extractive multi-modal summarization with multi-modal output

Extractive multi-modal summarization with multi-modal output (MSMO) aims to form a multi-modal summary by extracting sentences and images from the source multi-modal document. Existing works mostly formulate it as two sequence labeling problems by making individual sentence label prediction and image label prediction. This paper proposes a novel extractive multi-modal summarization method named DiffuMSMO, by directly generating the desired summary image representations with an ImageText2Image diffusion model and generating summary sentence representations with an ImageText2Text diffusion model, and then extracting images and sentences based on image representation matching and sentence representation matching. To bridge the gap between sentences and news images, DiffuMSMO fine-tunes a news-domain-specific CoCa model on a news image captioning dataset. A multi-modal Transformer encoder is optimized with a contrastive image matching loss, a contrastive sentence matching loss, and a contrastive sentence-image alignment loss for contextualized image and sentence encoding. Experiments on the MSMO dataset demonstrate that DiffuMSMO achieves competitive performance. It outperforms nearly all extractive and abstractive summarization baselines across all ROUGE and IP metrics. When compared with the strongest baseline, DiffuMSMO achieves statistically significant improvements on R-1 and R-L, while the baseline yields better results on R-2 and IP metrics. Ablation study shows effectiveness of each part of the model. The results demonstrate the great potential of adapting diffusion models for extractive MSMO.