A vision explainability method for image captioning using transformer decoder attention maps.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Image captioning models often function as black boxes, lacking transparency in their decision-making processes.
• There is a need for explainable AI (XAI) in vision-language models to build trust and reliability.

Purpose of the Study:

• To develop a novel explainable image captioning framework that integrates visual explainability.
• To address the transparency gap in current vision-language models.

Main Methods:

• Utilized a Convolutional Neural Network (CNN) encoder and a Transformer decoder architecture.
• Integrated attention-based heatmaps to provide visual explanations for generated captions.
• Evaluated performance and interpretability on the MS COCO dataset using standard metrics (BLEU, METEOR, CIDER, SPICE).

Main Results:

• The proposed framework offers transparency in the decision-making process of image captioning.
• Attention-based heatmaps effectively highlight visual features influencing caption generation.
• The method balances captioning quality with enhanced interpretability.

Conclusions:

• The developed framework increases trustworthiness and transparency in AI systems.
• This approach is suitable for critical applications in healthcare, education, security, and forecasting.
• It contributes to the advancement of explainable AI by bridging performance and interpretability in vision-language models.