Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Video

Updated: Jul 13, 2025

A Swin Transformer-Based Model for Thyroid Nodule Detection in Ultrasound Images
04:23

A Swin Transformer-Based Model for Thyroid Nodule Detection in Ultrasound Images

Published on: April 21, 2023

1.9K

What Makes for Hierarchical Vision Transformer?

Yuxin Fang, Xinggang Wang, Rui Wu

    IEEE Transactions on Pattern Analysis and Machine Intelligence
    |October 11, 2023
    PubMed
    Summary
    This summary is machine-generated.

    Related Concept Videos

    You might also read

    Related Articles

    Articles linked to this work by shared authors, journal, and citation graph.

    Sort by
    Same author

    Endogenous miR-204 Protects the Kidney against Chronic Injury in Hypertension and Diabetes.

    Journal of the American Society of Nephrology : JASN·2020
    Same author

    Serum proteomics analysis of feline mammary carcinoma based on label-free and PRM techniques.

    Journal of veterinary science·2020
    Same author

    Influence of Different Inactivation Methods on Severe Acute Respiratory Syndrome Coronavirus 2 RNA Copy Number.

    Journal of clinical microbiology·2020
    Same author

    Trends in high-impact papers in nursing research published from 2008 to 2018: A web of science-based bibliometric analysis.

    Journal of nursing management·2020
    Same author

    WRKY33-PIF4 loop is required for the regulation of H<sub>2</sub>O<sub>2</sub> homeostasis.

    Biochemical and biophysical research communications·2020
    Same author

    Protopanaxadiol ginsenoside Rd protects against NMDA receptor-mediated excitotoxicity by attenuating calcineurin-regulated DAPK1 activity.

    Scientific reports·2020

    Hierarchical Vision Transformers (ViT) can achieve top performance using a novel macro architecture. Replacing self-attention with linear layers in ViTs demonstrates strong image recognition and transfer learning capabilities.

    Area of Science:

    • Computer Science
    • Artificial Intelligence
    • Machine Learning

    Background:

    • Hierarchical Vision Transformers (ViT) with window-based self-attention achieve state-of-the-art results, challenging Convolutional Neural Networks (CNNs).
    • Self-attention is currently the standard for spatial information aggregation in hierarchical ViTs.

    Purpose of the Study:

    • To investigate if self-attention is essential for high performance in hierarchical ViTs.
    • To explore the impact of different cross-window communication methods.
    • To evaluate alternative methods for spatial information aggregation.

    Main Methods:

    • Replaced self-attention layers with linear mapping layers to create the TransLinear architecture.
    • Evaluated TransLinear on ImageNet-1K image recognition.

    More Related Videos

    Swin-PSAxialNet: An Efficient Multi-Organ Segmentation Technique
    04:48

    Swin-PSAxialNet: An Efficient Multi-Organ Segmentation Technique

    Published on: July 5, 2024

    439
    A Human Cerebral Organoid Model of Neural Cell Transplantation
    08:58

    A Human Cerebral Organoid Model of Neural Cell Transplantation

    Published on: July 21, 2023

    1.2K

    Related Experiment Videos

    Last Updated: Jul 13, 2025

    A Swin Transformer-Based Model for Thyroid Nodule Detection in Ultrasound Images
    04:23

    A Swin Transformer-Based Model for Thyroid Nodule Detection in Ultrasound Images

    Published on: April 21, 2023

    1.9K
    Swin-PSAxialNet: An Efficient Multi-Organ Segmentation Technique
    04:48

    Swin-PSAxialNet: An Efficient Multi-Organ Segmentation Technique

    Published on: July 5, 2024

    439
    A Human Cerebral Organoid Model of Neural Cell Transplantation
    08:58

    A Human Cerebral Organoid Model of Neural Cell Transplantation

    Published on: July 21, 2023

    1.2K
  • Assessed transfer learning capabilities on downstream tasks like object detection and instance segmentation.
  • Experimented with alternative content aggregation layers and cross-window communication strategies.

    • Main Results:

    • The TransLinear architecture achieved strong performance on ImageNet-1K image recognition.
    • TransLinear demonstrated competitive transfer learning properties on downstream dense prediction tasks.
    • The study found that the macro architecture, rather than specific aggregation layers or communication mechanisms, is the primary driver of performance in hierarchical ViTs.

    Conclusions:

    • Hierarchical ViT's macro architecture is the key factor for its strong performance, not necessarily self-attention.
    • Simple linear layers can effectively replace self-attention in ViTs, offering competitive results.
    • The findings suggest that the macro architecture of ViTs poses a significant challenge to traditional CNN paradigms.