EIM: An effective solution for improving multi-modal large language models.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Multi-modal large language models (LLMs) integrate vision and language, representing a significant advancement in AI.
• Current models often increase parameters by using larger encoders, complex fine-tuning, and transformation networks (STs).

Purpose of the Study:

• To propose EIM, an effective solution to improve multi-modal LLM performance by optimizing the training process.
• To reduce the need for additional parameters and structural modifications in multi-modal LLMs.

Main Methods:

• EIM incorporates improvements in the image encoder, semantic space transformation network (ST), and LLM components.
• The approach was validated on ClipCap (COCO Caption dataset) and extended to LLaMA-Adapter and LaVIN (ScienceQA dataset, MME benchmark).

Main Results:

• EIM application to a model on COCO Caption yielded a 1.75% performance increase with significantly fewer parameters.
• EIM-enhanced 7B models achieved competitive performance against 13B multi-modal LLMs on ScienceQA and MME.

Conclusions:

• EIM offers an effective method to enhance multi-modal LLM performance through training process optimization.
• This approach reduces computational costs and parameter requirements while improving capabilities.