Semantic clustering under resource constraints via Bayesian low-rank adaptation

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Traditional image clustering relies on visual similarity, limiting semantic understanding.
• Vision-Language Models (VLMs) enable semantics-aware clustering but struggle in low-resource scenarios due to noisy outputs.
• Small-scale VLMs exhibit semantic inconsistency, compromising text-based image representations.

Purpose of the Study:

• To develop a user-guided multimodal semantic clustering framework for low-resource, user-defined scenarios.
• To enhance the quality of text-based representations from small-scale VLMs.
• To improve the accuracy and robustness of image clustering with reduced computational overhead.

Main Methods:

• Utilizing a visual question answering VLM for image description generation.
• Employing a small-scale large language model for user-defined semantic classification.
• Implementing token-level confidence estimation for adaptive feature weighting and robust fusion.
• Introducing a Bayesian low-rank adaptation alignment module for cross-modal projection with structured sparsity.

Main Results:

• Consistent improvements in accuracy and robustness for user-defined semantic clustering tasks.
• Competitive performance on standard category clustering benchmarks.
• Substantially reduced computational overhead compared to large-scale VLMs.
• Demonstrated effectiveness in low-resource, user-defined semantic clustering scenarios.

Conclusions:

• The proposed framework offers a promising direction for controllable, interpretable, and scalable clustering in practical multimodal settings.
• It effectively mitigates issues of noisy generation and semantic inconsistency in small-scale VLMs.
• The method provides a viable solution for resource-constrained environments requiring semantic image clustering.