ProPy : Building interactive and efficient prompt pyramids upon CLIP for partially relevant video retrieval

Area of Science:

• Computer Science
• Artificial Intelligence
• Multimedia Retrieval

Background:

• Partially Relevant Video Retrieval (PRVR) is challenging due to segment-specific query relevance.
• Existing methods often focus on unimodal features, underutilizing powerful vision-language models like CLIP.
• There is a need for advanced models that can effectively process and retrieve video segments based on complex queries.

Purpose of the Study:

• To propose ProPy, a novel model architecturally adapted from CLIP for PRVR.
• To leverage multi-granularity event semantics for improved video retrieval accuracy.
• To address the limitations of existing unimodal approaches in PRVR.

Main Methods:

• ProPy utilizes a Prompt Pyramid framework to capture semantics at multiple granularity levels.
• A Layer-then-Segment two-stage event sampling strategy is employed to optimize memory usage and representation.
• An Ancestor-Descendant Interaction Mechanism facilitates dynamic semantic interactions among video events.

Main Results:

• ProPy achieves state-of-the-art (SOTA) performance on three public datasets for PRVR.
• The proposed model significantly outperforms previous state-of-the-art methods.
• The architectural innovations enable effective retrieval of videos based on partial relevance.

Conclusions:

• ProPy demonstrates the efficacy of adapting large-scale vision-language models for PRVR.
• The Prompt Pyramid and interaction mechanisms are key to capturing multi-granularity event semantics.
• This work advances the field of video retrieval by enabling more precise and efficient search capabilities.