Self-Guiding Multimodal LSTM-When We Do Not Have a Perfect Training Dataset for Image Captioning

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Real-world image-sentence datasets are often uncontrolled, imbalanced, and noisy.
• Existing multimodal recurrent neural network (RNN) captioning frameworks struggle with such data complexities.
• Accurate image captioning requires robust models that can handle diverse and imperfect training data.

Purpose of the Study:

• To propose a self-guiding multimodal LSTM (sgLSTM) model for image captioning.
• To address the challenges posed by uncontrolled, imbalanced, and noisy image-sentence datasets.
• To enhance the coupling between textual descriptions and visual content in image captioning models.

Main Methods:

• Collected and utilized the FlickrNYC dataset (306,165 images) with user-uploaded descriptions as ground truth.
• Developed a novel guiding textual feature extraction method using a multimodal LSTM (mLSTM) trained on strongly bonded image-description pairs.
• Integrated the guiding textual feature as additional input during sgLSTM training on the remaining data.

Main Results:

• The proposed sgLSTM model demonstrated superior performance compared to traditional state-of-the-art multimodal RNN captioning frameworks.
• The model successfully described key components of input images, indicating improved understanding of visual content.
• The guiding textual feature effectively addressed data imbalance and noise, enhancing captioning accuracy.

Conclusions:

• The sgLSTM model offers a robust solution for image captioning on challenging, real-world datasets.
• The novel guiding feature mechanism enhances the model's ability to tightly couple textual and visual information.
• This approach represents a significant advancement in multimodal learning for image description generation.