Monitoring the data quality of data streams using a two-step control scheme

Data-rich environments provide unprecedented opportunities for monitoring data quality. This article focuses on the quality of data streams. We use indicator variables to measure the six dimensions of data quality and a glitch index to indicate the poor level of quality. A two-step control scheme is proposed considering two relationships: the inter- and intra-correlation. In the first step, the Mahalanobis distance is applied to an χ²-type control chart to monitor the quality of a data stream. In the second step, a Shewhart control chart is built based on a weighted-sum statistic, which measures the quality of the whole process. The feasibility and effectiveness of the control scheme are illustrated through detailed simulation studies and one landslide example. The simulated results, considering the three cases of no correlation, low correlation, and high correlation, show that the proposed approach can detect the mean shift in multi-attribute data sensitively and robustly. The example, in which sensors are used to collect data on accelerations in Taiwan, demonstrates the superiority of our design over four traditional control charts, producing the closest type-I error to the given level and the highest power under the same type-I error.