Efficient GPT-4V level multimodal large language model for deployment on edge devices.

Area of Science:

• Artificial Intelligence
• Computer Science
• Machine Learning

Background:

• Multimodal large language models (LLMs) have advanced AI but are limited by size and computational cost.
• Current LLMs typically require cloud deployment, restricting use in resource-constrained or privacy-sensitive environments.

Purpose of the Study:

• To introduce MiniCPM-V, a novel family of efficient multimodal LLMs designed for edge devices.
• To demonstrate that high-performing multimodal models can be effectively deployed on local consumer hardware.

Main Methods:

• Developed MiniCPM-V models integrating architectural, training, and data advancements.
• Evaluated the 8B MiniCPM-V model against state-of-the-art models (GPT-4V, Gemini Pro, Claude 3) on 11 public benchmarks.

Main Results:

• The 8B MiniCPM-V model surpasses GPT-4V, Gemini Pro, and Claude 3 on multiple benchmarks.
• Achieved efficient performance on mobile phones, processing high-resolution images, robust OCR, low hallucination, and multilingual support (>30 languages).

Conclusions:

• MiniCPM-V represents a significant step towards democratizing powerful multimodal AI by enabling local operation on edge devices.
• Decreasing model sizes and increasing edge computation capacity facilitate advanced AI deployment in diverse real-world applications, including mobile and privacy-critical scenarios.