MPGB: Learning discriminative embeddings with multi-prototype and gradient balancing strategy for multi-modal 3D open world object detection

In recent years, extensive research has been conducted on the closed world 3D object detection. However, the closed-set scenario is not practical for the complex and dynamic real-world environment, especially for autonomous driving systems which require the ability to perceive and respond to various road traffic emergencies. This paper thoroughly investigates multi-modal 3D open world object detection. The primary challenges are unstructured nature (e.g. irregularity and sparsity) and data imbalance. To better capture the intra-class diversity and inter-class difference, we introduce the multi-prototype contrastive learning and a weighted cross-entropy loss. To handle long-tail data distribution problem, we utilize the multi-head structure for region proposal network (RPN) with rate and magnitude gradient balancing strategy. In addition, we incorporate prototypes as feature replay during incremental tasks to alleviate the catastrophic forgetting problem. Extensive experiments on the KITTI and Waymo datasets evidence that the proposed MPGB demonstrates superiority in recognizing both novel and known categories, compared to baselines. The code is available at https://github.com/zhanghaozhe23/MPGB.