A Survey of Geometric Optimization for Deep Learning: From Euclidean Space to Riemannian Manifold

Deep Learning (DL) has achieved remarkable success in tackling complex Artificial Intelligence tasks. The standard training of neural networks employs backpropagation to compute gradients and utilizes various optimization algorithms in the Euclidean space ĝ., ⁿ. However, this optimization process faces challenges, such as the local optimal issues and the problem of gradient vanishing and exploding. To address these problems, Riemannian optimization offers a powerful extension to solve optimization problems in deep learning. By incorporating the prior constraint structure and the metric information of the underlying geometric information, Riemannian optimization-based DL offers a more stable and reliable optimization process, as well as enhanced adaptability to complex data structures. This article presents a comprehensive survey of applying geometric optimization in DL, including the basic procedure of geometric optimization, various geometric optimizers, and some concepts of the Riemannian manifold. In addition, it investigates various applications of geometric optimization in different DL networks for diverse tasks and discusses typical public toolboxes that implement optimization on the manifold. This article also includes a performance comparison among different deep geometric optimization methods in image recognition scenarios. Finally, this article elaborates on future opportunities and challenges in this field.