Towards Balanced Representation Learning with Semantic Anchor Regularization

Representation learning refers to the process of learning meaningful and informative features from raw data, of which one good criterion is to attain intra-class compactness and inter-class separability in the semantic space. However,real-world data are always imbalanced and noisy. Existing methods such as prototype-based learning and contrastive learning are deeply bounded to the feature learning process and susceptible to imbalanced data distribution. In this paper, we disentangle the representation regularization from the feature learning process and propose a semantic anchor regularization (SAR) that is generated from predefined anchors. These anchors serve as an independent third-party measurement and are made semantic-aware by sharing the task head with feature learning. By controlling the separability between semantic anchors and pulling the learned representation to these semantic anchors, the intra-class compactness and inter-class separability can be intuitively achieved. In essence, SAR performs in the manner of visual-language alignment but is more flexible. Extensive results on classification, segmentation, long-tailed learning, and semi-supervised learning demonstrate the SAR’s effectiveness for different downstream tasks.