Going beyond still images to improve input variance resilience in multi-stream vision understanding models

Area of Science:

• Computer Vision
• Neuroscience
• Artificial Intelligence

Background:

• Traditional vision models primarily use spatial features from static images.
• Natural vision processes continuous spatiotemporal information, offering resilience.
• Existing video-understanding models have limited integration with image-understanding frameworks.

Purpose of the Study:

• To develop a brain-inspired vision model trained with videos.
• To investigate the impact of spatiotemporal features on model resilience.
• To bridge the gap between image and video understanding models.

Main Methods:

• Developed a novel brain-inspired vision model architecture.
• Trained the model using video data incorporating temporal features.
• Evaluated model resilience against various input alterations.

Main Results:

• Models trained on videos demonstrated superior resilience compared to those trained on static images.
• Inclusion of temporal features significantly boosted robustness.
• The brain-inspired approach showed enhanced adaptability to input variations.

Conclusions:

• Training with videos and temporal features is crucial for robust vision models.
• Brain-inspired models offer a promising direction for advanced computer vision.
• This study highlights the benefits of mimicking natural vision processing.