Dual CNN and ViT experts fusion for open set recognition

Area of Science:

• Computer Vision
• Deep Learning
• Artificial Intelligence

Background:

• Open set recognition (OSR) aims to classify known categories and detect unknown instances using deep neural networks.
• Current OSR methods predominantly use Convolutional Neural Networks (CNNs), which excel at local feature extraction.
• Vision Transformers (ViTs) offer superior global context understanding, presenting an opportunity for OSR enhancement.

Purpose of the Study:

• To develop an advanced OSR model by integrating the strengths of both CNNs and ViTs.
• To enhance the ability of deep learning models to differentiate between known and unknown data categories.
• To leverage multi-expert fusion for improved feature representation in OSR.

Main Methods:

• Proposed a multi-expert fusion network, the CNN and ViT Experts Fusion (CVEF) model.
• Integrated multiple Vision Transformer (ViT) and Convolutional Neural Network (CNN) experts.
• Developed an adaptive fusion mechanism to combine outputs from ViT and CNN experts for OSR.

Main Results:

• Demonstrated that ViT and CNN experts capture complementary image features.
• Achieved effective distinction between known and unknown categories through adaptive fusion.
• Validated the model's effectiveness and robustness on standard OSR benchmarks.

Conclusions:

• The CVEF model successfully integrates ViT and CNN architectures for superior OSR performance.
• The fusion approach enhances the model's ability to capture both global and local image context.
• The proposed method shows promise for broader applications in image processing and recognition.