MSSA: memory-driven and simplified scaled attention for enhanced image captioning

Image captioning is a key task in computer vision and natural language processing. It involves creating clear and accurate descriptions of what we see in images, helping to connect visuals with words in a meaningful way. This paper introduces MSSA (Memory-Driven and Simplified Scaled Attention), a novel framework for image captioning designed to enhance multimodal integration and caption generation. MSSA leverages Extended Multimodal Feature Extraction, incorporating a diverse range of features, including geometric features encoding spatial properties of bounding boxes, color features representing pixel intensity distributions in RGB space, texture features capturing local variations using Local Binary Patterns (LBP), edge features describing boundary structures via Canny edge detection, and frequency-domain features detecting orientation- and frequency-specific patterns through Gabor filters. This comprehensive feature set provides a richer understanding of complex visual scenes. The framework integrates two key mechanisms: Memory-Driven Attention (MDA) and Simplified Scaled Attention (SSA). MDA iteratively refines the alignment of visual and multimodal features using an LSTM-based memory mechanism, ensuring dynamic adaptation to contextually relevant image and textual elements. SSA generates context vectors by leveraging scaled dot-product attention, enabling efficient modeling of spatial, semantic, and contextual interactions while maintaining computational simplicity through the removal of complex gating mechanisms. Extensive experiments on the MSCOCO dataset demonstrate that MSSA outperforms state-of-the-art methods across several evaluation metrics. The proposed framework combines robust feature extraction with a simplified attention module, and we support the "streamlined" claim by reporting concrete efficiency evidence (Params/FP32 size, FLOPs, and inference latency) within our LSTM-based captioning pipeline, without implying a direct runtime advantage over Transformer-based captioning models. The codes and resources for MSSA are publicly available at: https://github.com/alamgirustc/MSSA .