Robust depth completion based on Semantic Aggregation

Abstract: Guided by information from RGB images, depth completion methods rebuild the dense depth from sparse depth input. However, the varying densities of valid pixels in sparse depth maps pose a significant challenge to the robustness of the completion model. To improve the robustness of depth completion, we propose a two-stage model called Semantic Aggregated Depth Completion (SADC) in this paper, comprising a coarse-grained completion stage and a fine-grained completion stage. In the coarse-grained completion stage, the Semantic Extraction Network (SEN) extracts RGB features and sends them to the Dynamic Semantic Aggregation (DSA) to predict the local semantic relationship (LSR) matrix. DSA aggregates the valid information based on the LSR matrix iteratively, resulting in coarse-grained completion results. In the fine-grained completion stage, SADC uses the Semantic Guidance Network (SGN) and Semantic Guidance Fusion (SGF) modules to refine the dense depth features from coarse-grained completion results by RGB features in multi-level and predict fine-grained completion results. We validate our method on NYU-v2 and KITTI with different valid pixel densities. The results demonstrate that SADC performs best results on benchmark tests and exhibits robustness to different densities without retraining. Graphical abstract: (Figure presented.)