Related Experiment Videos
HG-LMM: Unleashing High-Quality Pixel Grounding Capabilities in Frozen Large Multimodal Models
Summary
HG-LMM enhances Large Multimodal Models (LMMs) for pixel grounding without sacrificing conversational abilities. This novel approach achieves high-quality segmentation efficiently, outperforming existing methods.
Area of Science:
- Artificial Intelligence
- Computer Vision
- Natural Language Processing
Background:
- Large Multimodal Models (LMMs) excel at multimodal understanding and conversation.
- Fine-tuning LMMs for pixel grounding often degrades their conversational skills.
- Existing methods freeze LMMs and use heavy segmenters like SAM for grounding.
Purpose of the Study:
- To propose HG-LMM, a novel approach for high-quality pixel grounding using frozen LMMs.
- To preserve the inherent conversational capabilities of LMMs during grounding tasks.
- To improve segmentation accuracy and efficiency in multimodal tasks.
Main Methods:
- Developed HG-LMM with LLM-guided instance-aware feature generation (LIFG) and layer-wise detail and semantic injection (LDSI) modules.
- LIFG generates instance-aware feature maps using LMM text embeddings.
- LDSI injects detailed semantic information into feature maps for precise segmentation.
Main Results:
- HG-LMM achieved 12.6% higher segmentation accuracy than F-LMM without SAM on the PNG dataset.
- HG-LMM demonstrated comparable segmentation accuracy to F-LMM with SAM but was 2.7 times faster.
- The approach was validated across diverse tasks including referring expression segmentation and grounded conversation generation.
Conclusions:
- HG-LMM effectively leverages frozen LMM features for superior pixel grounding.
- The method successfully balances high-quality segmentation with preserved conversational abilities.
- HG-LMM offers a more efficient and capable solution for multimodal grounding tasks.