Ensemble deep learning with image captioning for visual pollution detection, classification, and reporting

Area of Science:

• Environmental Science
• Computer Science
• Urban Planning

Background:

• Rapid urban development in Saudi Arabia, including Saudi Vision 2030 initiatives, has led to environmental challenges like visual pollution (VP).
• Current VP reporting methods rely on manual data entry via an online application, which is error-prone.
• There is a need for automated, accurate systems to monitor and manage urban environmental quality.

Purpose of the Study:

• To propose an AI-driven framework for automated detection, classification, and reporting of visual pollution.
• To integrate deep learning models (YOLOv5, EfficientDet) with ensemble techniques for enhanced VP detection.
• To utilize Bootstrapping Language-Image Pre-training (BLIP) for automatic generation of image-based report descriptions.

Main Methods:

• Developed an AI framework combining YOLOv5 and EfficientDet models for visual pollution detection.
• Employed ensemble techniques to improve the performance of individual deep learning models.
• Integrated BLIP-2 (specifically BLIP2-Flan-T5-XL) for automatic image captioning to generate report descriptions.
• Utilized the
• Saudi Arabia Public Roads Visual Pollution Dataset
• for framework development and evaluation.

Main Results:

• The ensemble approach achieved a Mean Average Precision (mAP) of 0.95, recall of 0.95, precision of 0.91, and F1 score of 0.93.
• Individual models were surpassed by the ensemble method in visual pollution detection.
• The BLIP2-Flan-T5-XL model demonstrated 80% accuracy in generating descriptive text for urban images based on human evaluation.

Conclusions:

• The proposed AI framework effectively automates visual pollution detection and reporting, significantly reducing errors associated with manual entry.
• The integration of deep learning and image captioning facilitates the creation of citizen-monitored reports, improving urban management.
• This AI-driven approach contributes to enhancing the quality of life and promoting sustainable urban development in Saudi cities.