Harnessing Knowledge From Pretrained VLMs for Unsupervised Person Search

Person search is a unified task that includes the subtasks of pedestrian detection and re-identification (re-ID). It is expensive to label pedestrian bounding boxes and person identities for training. Purely unsupervised (US) person search is more practical for real-world situations. However, it is difficult to obtain accurate pseudo-IDs from low-quality pseudo-boxes, which brings a new challenge. To address this issue, we propose FMUPS, a novel method that leverages semantic information to produce reliable pseudo-labels. Semantic representations, particularly from vision-language models (VLMs), provide clear and interpretable guidance, reducing noise caused by background disturbances during pseudo-label extraction. Despite their advantages, VLM-generated pseudo-boxes often suffer from poor alignment with person regions and misclassification of other objects as people, adversely affecting the re-ID task. To overcome these issues, we introduce an anti-bbox-noise re-ID loss that not only alleviates the above localization and classification noises but also helps to acquire effective re-ID features. In addition, we propose a CLIP ID labeler, which exploits text-image alignment capabilities of VLMs to generate pseudo-IDs based on our predefined attributes and iteratively refines them using prior knowledge of person search. The experimental results on two typical benchmarks, CUHK-SYSU and PRW, demonstrate the effectiveness of our method; in particular, we outperform some previous fully and weakly supervised (WS) methods.