Pairwise CNN-Transformer Features for Human-Object Interaction Detection

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Human-object interaction (HOI) detection is crucial for computers to understand scene semantics.
• Existing two-stage HOI methods excel at feature quality but lack context.
• One-stage transformer-based methods capture global context but miss object detection benefits.

Purpose of the Study:

• To propose a novel two-stage method that integrates the strengths of both CNN and Transformer approaches for HOI detection.
• To enhance human-object pair representations by fusing CNN and Transformer features.
• To improve the contextual understanding in HOI detection models.

Main Methods:

• The Pairwise Convolutional neural network (CNN)-Transformer (PCT) model is proposed as a two-stage approach.
• Pairwise CNN features are extracted from a CNN backbone and fused with pairwise Transformer features.
• Global features from the Transformer are utilized to provide contextual cues.

Main Results:

• The fusion of CNN and Transformer features results in enhanced pairwise representations superior to individual feature types.
• Experimental comparisons demonstrate that CNN features retain a significant advantage in HOI detection.
• The PCT model achieves competitive performance compared to state-of-the-art methods on HICO-DET and V-COCO datasets.

Conclusions:

• The proposed PCT model effectively combines object detection capabilities with rich contextual information for HOI detection.
• Integrating CNN and Transformer features offers a synergistic advantage for HOI detection tasks.
• The study highlights the continued relevance of CNN features in advanced HOI detection architectures.