From Wrist to Finger: Hand Pose Tracking Using Ring-Watch Wearables

Hand pose tracking is essential for advancing applications in human-computer interaction. Current approaches, such as vision-based systems and wearable devices, face limitations in portability, usability, and practicality. This paper proposes a novel multimodal hand pose tracking framework that integrates data from an IMU-equipped ring and EMG sensors embedded in a wrist-worn device. By leveraging the complementary strengths of motion dynamics and muscle activity, our deep learning-based sensor fusion approach achieves precise 3D hand pose reconstruction. We fused multichannel data using a transformer-based model incorporating time encoding and cross-modal attention mechanisms. We also designed weighted loss function designed to optimize spatial, kinematic, and anatomical accuracy. Experimental validation using a custom dataset of 19 gestures performed by 10 participants demonstrates robust performance, with an average MPJPE of 0.750 cm and joint angle differences of 6.815° for cross-user evaluation.