Vocabulary-Free Image Classification and Semantic Segmentation.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Large vision-language models (VLMs) excel at image classification and segmentation but require predefined category vocabularies.
• This limitation is impractical for real-world scenarios with unknown or evolving semantic contexts.
• Existing methods struggle with the vastness of unconstrained semantic spaces, including fine-grained categories.

Purpose of the Study:

• To introduce the Vocabulary-free Image Classification (VIC) task, enabling classification into an unconstrained semantic space.
• To propose Category Search from External Databases (CaSED), a training-free method for VIC.
• To extend CaSED for vocabulary-free semantic segmentation.

Main Methods:

• CaSED leverages pre-trained VLMs and external databases to identify candidate categories from semantically similar captions.
• Images are classified by matching them to the best candidate category using the VLM.
• CaSED is applied locally for vocabulary-free semantic segmentation by classifying image regions.

Main Results:

• CaSED effectively performs image classification without a predefined vocabulary.
• CaSED variants achieve state-of-the-art performance on classification and semantic segmentation benchmarks.
• The proposed methods significantly outperform more complex VLMs while using fewer parameters.

Conclusions:

• Vocabulary-free image classification and segmentation are feasible and effective.
• CaSED offers a powerful and parameter-efficient solution for unconstrained semantic understanding.
• This work opens new avenues for applying VLMs in dynamic and open-ended environments.