FASSVid: Fast and Accurate Semantic Segmentation for Video Sequences

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Current real-time semantic segmentation methods often neglect temporal information in video sequences, which is crucial for real-world applications.
• Processing video frames sequentially without considering temporal context limits both speed and accuracy.

Purpose of the Study:

• To address the limitations of existing methods by incorporating temporal information into real-time semantic segmentation.
• To develop a network that leverages past frames and predictions to improve current frame segmentation accuracy.
• To introduce a new dataset, CityscapesVid, for benchmarking semantic video segmentation networks.

Main Methods:

• The proposed FASSVid network exploits temporal information from previous input frames and network outputs.
• A specialized module generates feature maps highlighting changes between frames.
• Previous network outputs are integrated into all decoder stages to enhance attention to relevant features.

Main Results:

• FASSVid demonstrates improved mean Intersection over Union (mIoU) accuracy compared to non-sequential baseline models.
• The network achieves state-of-the-art inference speeds, running at 114.9 FPS on a GTX 1080Ti and 31 FPS on a Jetson Nano.
• FASSVid presents competitive mIoU results with significantly fewer computations than other lightweight networks, achieving 71% mIoU on the CityscapesVid dataset.

Conclusions:

• FASSVid effectively utilizes temporal information for enhanced real-time semantic video segmentation.
• The method offers a superior balance of accuracy and inference speed, making it suitable for resource-constrained environments.
• The introduction of CityscapesVid provides a standardized benchmark for future research in this domain.