Interactive Target Element Selection in Infographics Via Scene Tree Modeling

Accurately detecting infographic elements is essential for infographic understanding and generation. However, high-performance detection models rely on large-scale training datasets with precise bounding-box annotations for infographic elements, which are time-consuming to create manually. Although fine-tuned object detection models can predict candidate bounding boxes, they often include non-target elements to be filtered out and therefore require efficient target element selection. To address this challenge, we develop a human-in-the-loop tool, EleDetective, that combines automatic selection, coordinated visualizations, and interactive refinement. The key idea is to represent an infographic as a scene tree that captures the region hierarchy. Based on this representation, we formulate target element selection as a bilevel optimization problem to recover missed elements and remove misselected elements. Since automatic selection is not always perfect, we introduce a treemap-based grid visualization and a scene tree visualization to help users explore selection results and identify potential errors. Users then correct the identified errors, and their corrections are propagated across infographics based on query-sensitive similarity, improving the quality of selection at scale. Quantitative evaluation and two use cases demonstrate the effectiveness of our method.