Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysis

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Large models exhibit In-Context Learning (ICL) capabilities, proven effective in Natural Language Processing (NLP).
• Large Multimodal Models (LMMs) are emerging with ICL, but demonstration configuration for multimodal ICL is underexplored.
• Controllable In-Context Examples (ICEs) offer efficient analysis of LMM inference.

Purpose of the Study:

• To comprehensively investigate multimodal in-context learning for image captioning.
• To explore demonstration configuration strategies (shot number, image retrieval, caption assignment) and analyze LMM attention characteristics.
• To provide dual perspectives on multimodal ICL in LMMs through external and internal analysis.

Main Methods:

• External investigation of demonstration configuration strategies: shot number, image retrieval, and caption assignment.
• Internal analysis of LMM attention characteristics using newly developed metrics.
• Auxiliary experiments on attention-driven acceleration/compression and comparison of LMMs based on pre-training data features.

Main Results:

• Demonstration configuration strategies significantly impact LMM performance in image captioning.
• Analysis of LMM attention characteristics reveals typical behavioral patterns.
• Identified differences in LMM performance are explainable by pre-training data features.

Conclusions:

• Combining external and internal analysis provides a dual perspective for understanding multimodal ICL in LMMs.
• Newly proposed metrics and analytical methods can be applied to broader research on large models.
• Optimizing ICEs configuration is crucial for enhancing LMM capabilities in multimodal tasks.