Interpretable Neural Networks for Video Separation: Deep Unfolding RPCA With Foreground Masking.

Area of Science:

• Computer Vision
• Machine Learning
• Signal Processing

Background:

• Traditional neural networks struggle with simultaneous background subtraction and foreground detection.
• Robust Principal Component Analysis (RPCA) is effective but lacks deep learning integration.
• Existing deep unfolding RPCA networks do not explicitly formulate foreground masks.

Purpose of the Study:

• To introduce two deep unfolding neural networks (ROMAN-S and ROMAN-R) for simultaneous video background subtraction and foreground detection.
• To leverage domain knowledge from masked RPCA within a deep learning framework.
• To develop lightweight, interpretable networks trainable on limited data.

Main Methods:

• Developed ROMAN-S and ROMAN-R by mapping Alternating Direction Method of Multipliers (ADMM) iterations to convolutional layers.
• Incorporated masked RPCA to decompose video into low-rank (background) and sparse (foreground mask) components.
• ROMAN-S uses l1-l1 minimization for temporal mask correlation; ROMAN-R employs dictionary learning and reweighted l1-l1 minimization for enhanced foreground detection.

Main Results:

• Both ROMAN-S and ROMAN-R outperform other deep unfolding networks and untrained optimization algorithms.
• ROMAN-R achieves competitive performance with 3D U-Net for foreground detection.
• ROMAN-R provides video backgrounds and requires significantly fewer training parameters and smaller datasets compared to U-Net.

Conclusions:

• Deep unfolding networks integrating masked RPCA offer an effective approach for video background subtraction and foreground detection.
• ROMAN-R presents a highly efficient alternative to U-Net, balancing performance with reduced computational and data requirements.
• The proposed models demonstrate strong generalization capabilities on unseen video clips.