Adaptive Latent Graph Representation Learning for Image-Text Matching.

Area of Science:

• Computer Vision
• Natural Language Processing
• Machine Learning

Background:

• Image-text matching faces challenges due to the modality gap.
• Existing methods modeling entity relationships are susceptible to irrelevant visual and textual information.
• Distractions in entity relationships hinder the learning of effective common embedding spaces.

Purpose of the Study:

• To propose an adaptive latent graph representation learning method for image-text matching.
• To reduce distractions from irrelevant entities in images and noisy words in text.
• To narrow the modality gap and improve matching performance.

Main Methods:

• Utilized an improved graph variational autoencoder to disentangle distracting factors from latent relationship factors.
• Jointly learned latent textual graph representations, latent visual graph representations, and a visual-textual graph embedding space.
• Introduced an adaptive cross-attention mechanism for feature attending on latent graph representations across modalities.

Main Results:

• Demonstrated significant effectiveness of the proposed method on the Flickr30K and COCO datasets.
• The adaptive latent graph approach successfully reduced distractions from irrelevant visual and textual elements.
• The adaptive cross-attention mechanism further enhanced feature alignment between image and text modalities.

Conclusions:

• The proposed adaptive latent graph representation learning method effectively addresses distractions in image-text matching.
• The method successfully narrows the modality gap, leading to improved matching performance.
• This approach offers a promising direction for future research in cross-modal retrieval and understanding.