Fine-Grained Video Retrieval With Scene Sketches.

Area of Science:

• Computer Science
• Artificial Intelligence
• Multimedia Retrieval

Background:

• Sketch-based video retrieval (SBVR) is intuitive but lacks fine-grained scene content accuracy.
• Existing methods struggle with precise retrieval based on detailed scene layouts and object characteristics.

Purpose of the Study:

• To introduce and address the challenge of fine-grained scene-level SBVR.
• To enable accurate video retrieval using detailed storyboard sketches of scene layout and object attributes.

Main Methods:

• Developed a novel deep learning architecture, Sketch Query Graph Convolutional Network (SQ-GCN).
• Constructed the SketchVideo dataset, featuring scene-level sketch-video pairs.
• Employed adaptive video frame sampling and constructed appearance/category graphs for alignment.

Main Results:

• The proposed SQ-GCN architecture significantly improves fine-grained scene-level SBVR.
• The SketchVideo dataset facilitates research in this specialized retrieval task.
• Experimental results demonstrate superior performance compared to state-of-the-art fine-grained retrieval methods.

Conclusions:

• The developed framework offers a robust solution for fine-grained scene-level SBVR.
• The combination of the SketchVideo dataset and SQ-GCN advances the field of sketch-based video retrieval.
• The approach effectively handles cross-modal alignment between detailed sketches and video content.