KEMoS: A knowledge-enhanced multi-modal summarizing framework for Chinese online meetings

Area of Science:

• Natural Language Processing
• Artificial Intelligence
• Information Retrieval

Background:

• The surge in online meetings generates vast data, necessitating efficient summarization.
• Existing models often neglect multi-modal information and semantic offsetting during summarization.

Purpose of the Study:

• To develop a knowledge-enhanced multi-modal framework for accurate and fast online meeting summarization.
• To address limitations in current models regarding multi-modal data utilization and information offsetting.

Main Methods:

• Constructed a three-layer multi-modal meeting knowledge graph for comprehensive information integration.
• Employed a topic-based hierarchical clustering approach considering information entropy and difference.
• Devised a multi-modal enhanced encoding strategy with cross-modal encoders and knowledge graph embeddings.
• Designed a topic-enhanced decoding strategy to mitigate semantic offsetting in generated summaries.

Main Results:

• The proposed framework consistently outperforms state-of-the-art solutions on a Chinese meeting dataset.
• Achieved ROUGE-1, ROUGE-2, and ROUGE-L scores of 49.98%, 21.03%, and 32.03% respectively.
• Demonstrated effective integration of multi-modal information and topic evolution for improved summarization.

Conclusions:

• The knowledge-enhanced multi-modal framework offers a significant advancement in meeting summarization.
• The approach effectively leverages multi-modal data and topic information to produce accurate and coherent summaries.
• This work provides a robust solution for the growing need for efficient online meeting summarization services.