OW-CLIP: Data-Efficient Visual Supervision for Open-World Object Detection via Human-AI Collaboration

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Open-world object detection (OWOD) requires continuous model adaptation for emerging objects, but current methods are data-hungry and prone to overfitting.
• Existing OWOD approaches often necessitate extensive crowdsourced annotations and inflexible model architecture modifications.

Purpose of the Study:

• To introduce OW-CLIP, a visual analytics system designed for data-efficient incremental training in OWOD.
• To address limitations of existing OWOD methods, including data requirements, partial feature overfitting, and model flexibility.

Main Methods:

• OW-CLIP utilizes plug-and-play multimodal prompt tuning and a novel Crop-Smoothing technique to mitigate overfitting.
• Dual-modal data refinement methods leverage large language models and cross-modal similarity for data generation and filtering.
• A visualization interface facilitates user exploration and high-quality annotation, including feature phrases and differentiated images.

Main Results:

• OW-CLIP achieves 89% of state-of-the-art performance using only 3.8% self-generated data.
• The system outperforms state-of-the-art methods when trained with equivalent data volumes.
• A case study confirmed the method's effectiveness and the enhanced annotation quality provided by the visualization system.

Conclusions:

• OW-CLIP offers a data-efficient and effective solution for open-world object detection model training.
• The system successfully mitigates partial feature overfitting and reduces reliance on large annotated datasets.
• The integrated visualization tool enhances annotation quality and user interaction in the OWOD pipeline.