Task-oriented Multimodal Semantic Communication Based on Double Channel State Estimation

Multimodal Semantic Communication (MSC) has emerged as a novel communication paradigm. Although MSC has great potential to improve communication efficiency and reliability by integrating heterogeneous data modalities, including text, images, and audio, existing MSC methods often assume perfect Channel State Information (CSI) and static channel conditions, severely limiting their applicability in highly dynamic real-world environments. Furthermore, they still primarily address single-modal tasks and lack robust mechanisms for handling complex multimodal tasks, such as audio-visual event localization over streaming data. These limitations hinder their ability to meet the requirements of task-specific applications in diverse scenarios. To address these challenges, we propose a task-oriented MSC framework with double channel state estimation tailored for audio-visual event localization. The framework leverages digital pilot codes and channel modules to track and adjust the state of analog channels under varying channel conditions and employs Euler-based multimodal semantic encoding and decoding that account for time-frequency characteristics based on the dynamic channel state. Extensive experimental results demonstrate that the proposed framework outperforms several baseline methods in terms of event localization accuracy at the same Signal-to-Noise Ratio (SNR), highlighting its advantage in semantic communication quality.