A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective.

Area of Science:

• Computer Vision
• Graph Representation Learning
• Deep Learning

Background:

• Graph Neural Networks (GNNs) excel in graph representation learning, advancing fields like data mining, computer vision, and NLP.
• Graph Transformers integrate graph structures into Transformer architectures, addressing limitations of local aggregation and rigid inductive biases.

Purpose of the Study:

• To provide a comprehensive review of Graph Neural Networks (GNNs) and graph Transformers in computer vision.
• To examine applications from a task-oriented perspective, categorized by input data modality and specific vision tasks.

Main Methods:

• Categorizing GNN and graph Transformer applications in computer vision into five input data modalities: 2D images, videos, 3D data, vision + language, and medical images.
• Further subdividing applications within each category by specific vision tasks.
• Analyzing task definitions, challenges, representative approaches, insights, limitations, and future research directions.

Main Results:

• A structured taxonomy of GNN and graph Transformer applications in computer vision based on input data and tasks.
• In-depth coverage of representative GNN-based and graph Transformer approaches for various computer vision tasks.
• Discussion on the performance, limitations, and future potential of these graph-based models in computer vision.

Conclusions:

• Graph Neural Networks and graph Transformers offer powerful tools for diverse computer vision tasks.
• A task-oriented, modality-specific review provides valuable insights into current capabilities and future research avenues.
• The paper serves as a comprehensive resource for understanding and advancing graph-based methods in computer vision.