mm3DFace: Nonintrusive 3D Facial Reconstruction Leveraging mmWave Signals

Recent years have witnessed the emerging market of 3D facial reconstruction that supports numerous face-driven scenarios including modeling in virtual reality (VR), human-computer interaction, and affective computing applications. Current mainstream approaches rely on vision for 3D facial reconstruction, which may encounter privacy concerns and suffer from obstruction scenes and bad lighting conditions. In this paper, we present a nonintrusive 3D facial reconstruction system, mm3DFace, which leverages a millimeter wave (mmWave) radar to reconstruct 3D human faces that continuously express facial expressions in a privacy-preserving and passive manner. Based on the pre-processed mmWave signals, mm3DFace first extracts facial geometric features that capture subtle changes in facial expressions through a ConvNeXt model with triple loss embedding. Then, mm3DFace derives distance and orientation-robust facial shapes with 68 facial landmarks using region-divided affine transformation. mm3DFace next reconstructs facial expressions through a designed regional amplification method and finally generates 3D facial avatars that continuously express facial expressions. Extensive experiments involving 15 participants in real-world environments show that mm3DFace can accurately track 68 facial landmarks with 3.94% normalized mean error, 2.30mm mean absolute error, and 4.10mm 3D-mean absolute error, which is effective and practical in real-world 3D facial reconstruction.