Dual CNN and ViT experts fusion for open set recognition

Open set recognition (OSR) aims to enable deep neural networks to accurately classify known categories while identifying instances belonging to unknown categories. Existing OSR methods often adopt discriminative approaches, with convolutional neural networks (CNNs) serving as the core architecture. However, vision transformers (ViTs), transformer-based architectures developed for computer vision, have demonstrated superior capabilities in capturing global contextual information compared to CNNs. This study proposes a multi-expert fusion network that integrates ViT and CNN architectures for OSR, termed the CNN and ViT Experts Fusion (CVEF) model. CVEF integrates multiple ViT and CNN experts, enabling the capture of both global and local image features while adaptively fusing their outputs to distinguish between known and unknown categories. Analysis reveals that ViT and CNN experts attend to distinct aspects of images. Extensive experiments on standard OSR benchmarks demonstrate the effectiveness and robustness of the model. The proposed approach holds potential for broader applications in image processing and recognition tasks. The code for this work is available at https://github.com/xulaupuz/CVEF.