Vision Enhancing LLMs: Empowering Multimodal Knowledge Storage and Sharing in LLMs

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Multimodal large language models (MLLMs) primarily map visual data to language spaces, utilizing LLM strengths for generation.
• Current MLLMs often overlook visual knowledge's potential to enhance core LLM capabilities, a concept termed 'Vision Enhancing LLMs'.

Purpose of the Study:

• To propose MKS2, a method for enhancing LLMs via Multimodal Knowledge Storage and Sharing.
• To improve LLM reasoning, particularly in physical and commonsense knowledge domains, by integrating visual information.

Main Methods:

• Introducing Modular Visual Memory (MVM) integrated within LLM internal blocks for efficient open-world visual information storage.
• Implementing a soft Mixture of Multimodal Experts (MoMEs) architecture to facilitate multimodal knowledge collaboration during text generation.

Main Results:

• MKS2 significantly boosts LLM reasoning abilities in tasks requiring physical and commonsense knowledge.
• The proposed approach achieves competitive performance on established image-text understanding multimodal benchmarks.

Conclusions:

• MKS2 effectively enhances LLMs by enabling them to leverage visual knowledge for improved reasoning and task performance.
• The MKS2 framework offers a promising direction for developing more capable and versatile Vision Enhancing LLMs.