Exploring Multimodal Knowledge for Image Compression via Large Foundation Models.

Area of Science:

• Computer Vision
• Machine Learning
• Information Theory

Background:

• Large foundation models store extensive multimodal knowledge, enabling machine intelligence for various tasks.
• The application of this knowledge to enhance image compression, particularly at ultra-low bitrates, remains underexplored.
• Ultra-low bitrate compression requires incorporating external knowledge due to sparse encoded representations.

Purpose of the Study:

• To harness multimodal knowledge from foundation models for ultra-low bitrate image compression.
• To propose a novel method, Multimodal Knowledge-aware Image Compression (MKIC), for efficient image coding.
• To improve the accuracy and compactness of semantic and visual information representation.

Main Methods:

• Integrating natural visual knowledge and human language knowledge into the compression framework.
• Utilizing a novel Alternating Rate-Distortion Optimization for semantic text representation extraction.
• Extracting local feature maps for visual details and integrating multimodal representations into a generative foundation model.

Main Results:

• The proposed MKIC method achieves superior comprehensive performance compared to existing methods.
• Demonstrates significant potential for achieving high-quality image reconstruction at ultra-low bitrates.
• Effectively stores shared patterns and sparse unique features by incorporating world knowledge.

Conclusions:

• Multimodal knowledge integration is crucial for advancing ultra-low bitrate image compression.
• MKIC offers a promising direction for learned image coding by leveraging decoupled knowledge from foundation models.
• The method enhances both the efficiency and quality of image compression in challenging low-bitrate scenarios.