DKMap: Interactive Exploration of Vision-Language Alignment in Multimodal Embeddings via Dynamic Kernel Enhanced...

Area of Science:

• Computer Science
• Artificial Intelligence
• Data Visualization

Background:

• Multimodal embeddings are vital for AI tasks like generative model evaluation.
• Existing dimensionality reduction (DR) methods struggle with cross-modal alignment visualization.
• Problems include point occlusion, inaccurate contour maps, and lack of multi-scale exploration.

Purpose of the Study:

• To introduce DKMap, a novel DR visualization technique for multimodal embeddings.
• To address limitations of existing DR methods in visualizing cross-modal alignment.
• To enable accurate, multi-scale interactive exploration of high-dimensional multimodal data.

Main Methods:

• Introduced kernel regression supervised t-SNE integrating contour mapping into projection learning.
• Integrated validation constrained α refinement with a generalized t-kernel and quad-tree multi-resolution.
• Developed DKMap as a multi-platform tool with a web system and Python package.

Main Results:

• DKMap accurately maps cross-modal alignment metrics, outperforming baseline DR techniques.
• Demonstrated superiority in handling divergent metrics and over-aggregation issues.
• Validated generalizability and scalability on large-scale datasets (million-scale and billion-scale).

Conclusions:

• DKMap offers a superior method for visualizing and exploring multimodal embeddings.
• The technique enhances understanding of vision-language alignment in AI.
• DKMap is suitable for diverse applications, including model evaluation and large dataset analysis.