Video reconstruction through dynamic scattering media based on physics-informed spatio-temporal transformer.

Area of Science:

• Computational Imaging
• Deep Learning
• Optics

Background:

• Video reconstruction through dynamic scattering media is challenging due to complex temporal dynamics and physics.
• Traditional methods struggle with scattering phenomena, limiting reconstruction quality and applicability.

Purpose of the Study:

• To present PISTA (Physics-Informed Spatio-Temporal Transformer Architecture), a novel deep learning framework.
• To improve video reconstruction in dynamic scattering environments by addressing limitations in temporal correlation modeling, physics constraints, and parameter estimation.

Main Methods:

• PISTA integrates a physics-informed encoder (enforcing energy conservation, temporal causality, reciprocity) with a spatio-temporal attention module.
• A parameter estimation network adaptively learns time-varying scattering coefficients.
• The framework utilizes a Transformer Architecture for enhanced spatio-temporal dependency capture.

Main Results:

• PISTA demonstrates notable improvements over conventional CNN- and transformer-based baselines.
• Validation on synthetic data and the OTIS real turbulence dataset confirms effectiveness.
• The framework achieves physically consistent and data-efficient video reconstruction.

Conclusions:

• PISTA offers a robust solution for video reconstruction in challenging dynamic scattering environments.
• The integration of physics-informed principles and attention mechanisms enhances reconstruction accuracy and adaptability.
• This work advances computational imaging for applications in autonomous driving and biomedical imaging.