视觉视觉的MetaFormer基线
IEEE transactions on pattern analysis and machine intelligence
|November 1, 2023
概括
MetaFormer 架构通过基本的代币混合器实现了强的性能,证明了其坚实的基础. 即使是像身份映射或随机矩阵这样的简单混合器也会产生高精度,而像CAFormer这样的高级模型则会创下新的记录.
科学领域:
- 计算机视觉 计算机视觉
- 深度学习架构 深度学习架构
- 神经网络设计 神经网络设计
背景情况:
- 该MetaFormer架构,一个抽象的变压器,已经显示在实现竞争性表现的承诺.
- 以前的研究集中在这些架构中的代币混合器设计上.
- 这项研究将重点转移到MetaFormer框架本身的内在能力.
研究的目的:
- 探索MetaFormer架构的性能潜力,独立于复杂的令牌混合器设计.
- 用基本和传统的令牌混合器验证MetaFormer的有效性.
- 为了介绍和评估一个新的激活功能,StarReLU.
主要方法:
- 使用简单的令牌混合器开发了基线MetaFormer模型:身份映射 (IdentityFormer) 和随机矩阵 (RandFormer).
- 实现了使用深度可分离卷积的ConvFormer,与ConvNeXt.进行比较.
- 通过将深度可分离的卷积和自我注意力结合起来,创建了CAFormer.
- 作为激活功能,引入并测试了Squared ReLU的变体StarReLU.
主要成果:
- IdentityFormer在ImageNet-1K上实现了超过80%的准确性,建立了一个坚实的性能基线.
- RandFormer以超过81%的准确度超过了IdentityFormer,展示了MetaFormer与任意混合器的兼容性.
- 在ConvNet-1K上,ConvFormer表现优于ConvNeXt,CAFormer在ImageNet-1K上实现了85.5%准确度的新纪录.
- 与GELU相比,StarReLU减少了71%的激活FLOP,同时提高了性能.
结论:
- MetaFormer提供了一个强大的框架,可以确保高性能下界,即使使用最小的令牌混合器.
- 该架构展示了多功能性,与多种甚至随机的代币混合器表现良好.
- 当MetaFormer与StarReLU等传统或新型组件相结合时,可以有效地实现最先进的结果.
相关概念视频
Depth Perception and Spatial Vision
678
Depth perception is the ability to perceive objects three-dimensionally. It relies on two types of cues: binocular and monocular. Binocular cues depend on the combination of images from both eyes and how the eyes work together. Since the eyes are in slightly different positions, each eye captures a slightly different image. This disparity between images, known as binocular disparity, helps the brain interpret depth. When the brain compares these images, it determines the distance to an object.
678
Vision
53.5K
Vision is the result of light being detected and transduced into neural signals by the retina of the eye. This information is then further analyzed and interpreted by the brain. First, light enters the front of the eye and is focused by the cornea and lens onto the retina—a thin sheet of neural tissue lining the back of the eye. Because of refraction through the convex lens of the eye, images are projected onto the retina upside-down and reversed.
53.5K
Parallel Processing
157
The brain processes sensory information rapidly due to parallel processing, which involves sending data across multiple neural pathways at the same time. This method allows the brain to manage various sensory qualities, such as shapes, colors, movements, and locations, all concurrently. For instance, when observing a forest landscape, the brain simultaneously processes the movement of leaves, the shapes of trees, the depth between them, and the various shades of green. This enables a quick and...
157


