A framework for efficient scientific diagram captioning using mixture-of-experts and low-rank adaptation

Generating captions for diagrams is a challenging task because of their complex structures, diverse visual elements, and domain-specific semantic content. Unlike natural images, diagrams require a detailed understanding of the interplay between visual and textual components. Models such as BLIP-2, MiniGPT-4, and LLaVA are well-suited for general multimodal tasks. However, when applied to the specific demands of diagram captioning, they require longer inference times, making them less practical for this specialized domain. To address these challenges, we propose a framework that integrates a Mixture of Experts (MoE) with Low-Rank Adaptation (LoRA), trained on the AI2D-Caption dataset-comprising expert annotated scientific diagrams from biology, astronomy, and engineering-to achieve resource-efficient and high-quality caption generation. The framework operates in two stages. During the pretraining stage, the model aligns visual and textual modalities into a unified multimodal representation. In the fine-tuning stage, this representation is dynamically routed to domain-specific experts fine-tuned with LoRA. The gating network selectively activates relevant experts. Our approach outperforms the zero-shot performance of existing vision-language models, achieving an average BLEU score of 0.42 and an average SPICE score of 0.38, while maintaining an efficient average inference time of 3 seconds. Our contribution lies in proposing a new benchmark for diagram captioning. Our model using fewer parameters is able to outperform competing models.