IMUFace: towards always-on 3D facial reconstruction via earphone inertial sensing

Facial expression reconstruction technology offers considerable potential in areas like human-computer interaction, affective computing, and virtual reality. To tackle the privacy challenges and environmental constraints inherent in traditional camera-based systems, researchers have recently introduced ear-worn devices as a viable solution. Nevertheless, these methods still demand enhancements, particularly in aspects such as design appeal and energy efficiency, to achieve broader applicability and practical use. This paper introduces a system called IMUFace. It uses inertial measurement units (IMUs) embedded in wireless earphones to detect subtle ear movements caused by facial muscle activities, allowing for covert and low-power facial reconstruction. A user study involving 12 participants was conducted, and a deep learning model named IMUTwinTrans was proposed. The results show that IMUFace can accurately predict users’ facial landmarks with a precision of 2.21 mm, using only five minutes of training data. The predicted landmarks can be utilized to reconstruct a three-dimensional facial model. IMUFace operates at a sampling rate of 30 Hz with a relatively low power consumption of 58 mW. The findings validate the feasibility of IMUFace and highlight potential directions for further research towards its practical adoption in mobile environments.