Materialized view maintenance in columnar storage for massive data analysis

Data-intensive computing becomes a buzz word nowadays, where constant data for current operational processing and historical data for massive analysis are often separated into two systems. How to keep the historical data for analysis (often in a materialized view manner) consistent with their data sources (often in the operational databases) is the main problem to be solved imperatively. In this paper, we proposed a novel method for data consistency maintenance between the data located in the two systems. Two basic operators (i.e., insertion and deletion) for consistency maintenance are provided as well as their implementations in the new environment of column-oriented storage on large-scale data analysis platform for efficient processing. Two data consistency models (i.e., eventual consistency model and timeline-based consistency model) are proposed to tradeoff data consistency for processing efficiency. Our extensive experimental evaluation also proves the efficiency and effectiveness of our proposed methods.