Style-Enhanced Transformer for Image Captioning in Construction Scenes.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Construction Management

Background:

• Image captioning is crucial for construction project intelligence and site activity monitoring.
• Existing image captioning models struggle with the complexity of construction scenes.
• A lack of specialized datasets and models for construction imagery hinders progress.

Purpose of the Study:

• To develop an advanced image captioning model tailored for complex construction environments.
• To improve the accuracy and relevance of generated captions for construction site activities.
• To enhance the intelligence and automation capabilities in construction project management.

Main Methods:

• A novel style-enhanced Transformer for image captioning (SETCAP) was proposed.
• Swin Transformer was used for grid feature extraction, incorporating style information via a style encoder.
• Style information was integrated into text features within the decoder for word-by-word sentence generation.
• A sentence style loss was added to align generated captions with the training data style.

Main Results:

• The proposed SETCAP model achieved significant improvements on both MSCOCO and MOCS datasets.
• SETCAP outperformed state-of-the-art methods, showing a 4.2% CIDEr score increase on MOCS.
• A 3.9% CIDEr score improvement was observed on the MSCOCO dataset, demonstrating broad applicability.

Conclusions:

• The SETCAP model effectively addresses the limitations of existing methods in complex construction scenes.
• Integrating style information enhances the performance of image captioning for specialized domains.
• The developed model offers a promising solution for intelligent construction management and site activity analysis.