Zero-Shot Visual Grounding via Cascade Semantic Prototype Learning

Compared to standard visual grounding (VG), zero-shot visual grounding (ZSVG) requires robust localization of novel image-query pairs unseen during training, creating a clear gap from supervised settings. Existing two-stage approaches rely too heavily on external knowledge from pre-trained vision-language models and underexplore in-context semantics, while current one-stage models fail to leverage region-level cues to enhance discriminative representation learning. In this paper, inspired by the information processing procedure of human beings, we propose a novel one-stage network, dubbed Cascade Semantic Prototype Learning (CaSePro), to tackle these limitations. Specifically, CaSePro builds two types of prototypes that capture prototypical multimodal representations for global contexts and local semantics, respectively. With global context prototypes, CaSePro explores context-related cues to enhance the coarsely fused global multimodal features, enabling the model to learn more comprehensive representations. With local semantic prototypes, CaSePro searches for region-level multimodal correspondences to augment the prototypical global patterns, thereby strengthening the cross-modal relationship between the query and visual representations. Moreover, to enhance the generalization ability of the proposed approach on sample-imbalanced benchmark datasets, CaSePro further incorporates an Adaptive Focal Loss that dynamically reweights hard and easy samples by accounting for the model's grounding uncertainty. Extensive experiments on ten benchmarks demonstrate that CaSePro outperforms all existing one-stage approaches and achieves performance comparable to state-of-the-art two-stage methods.