Switchable Novel Object Captioner.

Area of Science:

• Computer Vision
• Natural Language Processing
• Artificial Intelligence

Background:

• Image captioning models require extensive paired image-sentence data for training.
• Existing models struggle with novel objects and out-of-vocabulary terms in new domains.
• Describing novel objects without additional training data presents a significant challenge.

Purpose of the Study:

• Introduce and address the zero-shot novel object captioning task.
• Develop a method for generating image descriptions of novel objects without specific training sentences.
• Mimic human learning by using knowledge of similar objects to describe the unknown.

Main Methods:

• Construct a key-value object memory using object detection models, storing visual information and associated words.
• Employ proxy visual words from similar seen objects to handle novel objects and out-of-vocabulary terms.
• Propose a Switchable Long Short-Term Memory (LSTM) network that integrates object memory for sentence generation, featuring two modes: standard sentence generation and retrieval of proper nouns.

Main Results:

• The proposed Switchable LSTM effectively disentangles language generation from training objects.
• The model successfully describes novel objects without requiring any specific training sentences for them.
• Experiments on three large-scale datasets validate the method's capability in describing novel concepts.

Conclusions:

• The developed approach enables effective zero-shot novel object captioning.
• This method significantly advances the ability of AI to describe unseen objects in images.
• The technique offers a robust solution for handling domain shifts and novel vocabulary in image description tasks.