A bio-inspired positional embedding network for transformer-based models.

Area of Science:

• Computer Vision
• Neuroscience
• Artificial Intelligence

Background:

• Transformer networks have advanced vision models, but positional embeddings require further optimization.
• Positional embeddings are critical for distinguishing spatial information in vision models.

Purpose of the Study:

• To propose a novel positional embedding network inspired by human visual pathways.
• To enhance the performance of transformer-based vision models through biologically plausible spatial information capture.

Main Methods:

• A double-stream architecture modeling the dorsal pathway for spatial perception.
• Utilizing large zero-padding convolutions for local features and transformers for global features.
• Integrating dorsal and ventral pathway interactions for comprehensive spatial understanding.

Main Results:

• The proposed method significantly improves image classification performance across various datasets.
• Statistical analysis confirms the effectiveness and biological plausibility of the approach.
• The simple implementation leads to notable performance gains.

Conclusions:

• The biologically inspired positional embedding network offers a promising direction for advancing vision models.
• Modeling human visual pathways can lead to more effective and interpretable AI systems.
• This approach enhances spatial feature extraction in transformer-based vision architectures.