Beyond catastrophic forgetting: A continual learning-driven multi-modal fusion model for saliency prediction in dynamic scenes

Modeling and predicting human attention in audio-visual circumstances is crucial to understanding and simulating human visual attention mechanism in a real multi-modal environment. In practical scenarios, data collected from various scenes often arrives sequentially, resulting in continuous shifts in the data distribution. However, the majority of existing audio-visual saliency approaches are developed based on the assumption of complete data availability and a stationary data distribution. Consequently, this assumption makes them inevitably suffer from the issue of catastrophic forgetting. To address the aforementioned issue, we are the first to propose a novel continual learning-driven multi-modal fusion model (CL-MMF) for saliency prediction in dynamic scenes. Notably, this model is the first to introduce continual learning technique into audio-visual saliency prediction task. Its primary goal is to effectively mitigate catastrophic forgetting, a challenge that has been largely overlooked in this field. Furthermore, we are the first to extend the world model into a multi-modal architecture that enables the joint generation and replay of both visual and auditory modalities, enhancing multi-modal consistency and stability in continuous learning environments. Specifically, we first introduce a new multi-modal world model to capture task-specific audio-visual dynamic information in a Gaussian mixture latent space. Second, we devise an experience replay method to generate the previous task data, which effectively addresses the issue of catastrophic forgetting. Finally, we introduce task semantic labels into the CL-MMF model to align audio-visual consistency. Furthermore, we construct two new task-based benchmarks according to video content and dataset membership. Extensive experimental results on two benchmarks demonstrate the superiority of the proposed CL-MMF model, outperforming the naive combinations of previous state-of-the-art methods in saliency prediction and continual learning by approximately 8% in performance efficiency.