Investigating the intrinsic top-down dynamics of deep generative models

Area of Science:

• Computational neuroscience
• Deep learning
• Artificial intelligence

Background:

• Hierarchical generative models learn data distributions and may explain spontaneous brain activity.
• Deep Belief Networks (DBNs) are unsupervised, energy-based models learning hierarchical representations.
• Current theories link resting-state brain activity to top-down generative model dynamics.

Purpose of the Study:

• Investigate the generative dynamics of the iterative Deep Belief Network (iDBN).
• Explore methods to enhance data generation diversity in hierarchical models.
• Connect generative model dynamics to neurocognitive development theories.

Main Methods:

• Trained iterative Deep Belief Network (iDBN) models on handwritten digits and faces.
• Analyzed top-down sampling dynamics and state visitation.
• Experimented with initializing generation from "chimera states" (combined high-level features).
• Compared iDBN dynamics to a shallow Restricted Boltzmann Machine.

Main Results:

• Initializing iDBN sampling from "chimera states" increased the diversity of generated data prototypes.
• The iDBN exhibited richer top-down dynamics compared to shallow models.
• Generated samples supported continual learning via generative replay.
• Model dynamics were influenced by energy function shape, architecture depth, and data structure.

Conclusions:

• Iterative Deep Belief Networks (iDBNs) show enhanced generative capabilities through biased initial states.
• The study provides a computational framework linking generative models to brain function and development.
• Findings suggest potential for iDBNs in continual learning and understanding neural dynamics.