A communication-efficient distributed algorithm for online monitoring in big data systems

In the era of big data, the vast amounts of available information pose significant challenges to traditional monitoring algorithms. The sheer volume of data often results in its distribution across multiple machines, making full communication between these data points impractical due to prohibitive costs or privacy concerns. To address these challenges in constructing online monitoring statistics, we propose a communication-efficient distributed algorithm. Specifically, this algorithm transfers only summarized statistics between machines, rather than individual-level data. This approach not only mitigates the issue of large data volumes that cannot be centralized on a single machine but also overcomes the privacy concerns associated with direct data sharing. Furthermore, we introduce a divide-and-conquer scheme that enhances communication efficiency by focusing on the integration of gradient information for shift estimation. To further reduce communication costs, only vector-type information is exchanged, rather than matrix or more complex data. The proposed algorithm leads to an enhanced exponentially weighted moving average (EWMA) control chart, thereby improving its monitoring capabilities. The reliability of the method is rigorously validated. Finally, simulation results and a practical example illustrate the effectiveness and superiority of the proposed scheme.