A Conditional Random Fields Based Framework for Multiview Sequential Data Modeling

Multiview learning has gained much attention due to the increasing amount of multiview data which are collected from different sources or can be characterized by different types of features. How to properly handle view heterogeneity and balance information between views are the challenges in the domain of multiview learning, especially for sequential data. In this paper, we propose a multiview conditional random field model (multiview CRF) for modeling multiview sequential data, in which the model focuses on different views at a different moment with varying degrees. Especially, additional weight variables are introduced in the CRF through two different mechanisms which play a role of controlling the use of information from different views. Besides, a multiview decomposition network is designed to transform multiview observation at each moment into a unified multiview representation which is further used as an input of weight variable CRF. Variational inference with variance reduction technique is adopted for model training. We conduct experiments on two real-world datasets to compare the proposed methods with other related state-of-the-art methods. Experimental results and careful analysis show that both the weight variables and decomposition network contribute to the outstanding performance over other methods.