A spatiotemporal style transfer algorithm for dynamic visual stimulus generation

Antonino Greco ^1,2,3, Markus Siegel ^4,5,6,7

⁰Department of Neural Dynamics and Magnetoencephalography, Hertie Institute for Clinical Brain Research, University of Tübingen, Tübingen, Germany. antonino.greco@uni-tuebingen.de.

Nature Computational Science +

|

December 20, 2024

View abstract on PubMed

Summary

Researchers developed a new dynamic visual stimulus generation framework, the spatiotemporal style transfer (STST) algorithm, to create videos for vision research. This tool helps study how visual information is encoded by manipulating low-level and high-level features in stimuli.

Area Of Science

Computational Neuroscience
Computer Vision
Psychophysics

Background

Generating appropriate visual stimuli is crucial for testing hypotheses in biological and artificial visual systems.
Existing methods for dynamic visual stimulus generation are limited, hindering progress in vision research.

Purpose Of The Study

To introduce a novel framework for dynamic visual stimulus generation and manipulation.
To create stimuli that isolate low-level spatiotemporal features from high-level semantic information.
To investigate the encoding of visual information in both deep learning models and human observers.

Main Methods

Development and application of the spatiotemporal style transfer (STST) algorithm for video synthesis.
Generation of stimuli with preserved low-level features but removed high-level semantic content.
Independent spatiotemporal factorization of dynamic visual stimuli.
Probing a predictive coding deep network (PredNet) and testing human observers with generated stimuli.

Main Results

STST algorithm successfully generated dynamic stimuli with controlled feature content.
PredNet's next-frame predictions were unaffected by the omission of high-level semantic information.
Human observers confirmed the preservation of low-level features and absence of high-level information in STST stimuli.
Testing factorized stimuli revealed a spatial bias in visual information encoding by humans and deep models.

Conclusions

The STST algorithm is a versatile tool for generating dynamic visual stimuli in vision science.
The findings provide insights into how visual information, particularly dynamic aspects, is processed.
The study highlights potential differences and similarities in visual encoding between artificial and biological systems.