DS²: Handling data skew using data stealings over high-speed networks

Distributed in-memory computing systems have dramatic performance improvement over traditional disk-based systems, which makes them widely used in large-scale data processing applications. Unfortunately, uneven and unpredictable data distributions caused by data skew have a significant impact on the performance. In Spark, when data skew happens, some tasks will process much more data than other tasks and become the performance bottleneck. The traditional approaches to handling data skew are based on sampling and repartitioning, which incur additional overhead. In this paper, we divide data skew in distributed data processing systems into intra-node and inter-node skew. Based on data stealing, we proposed DS2 to handle both intra-node and inter-node data skew. It aims to improve the performance under data skew, without involving additional overhead. DS2 first balances the skewed data distribution in the local and then handles the inter-node skew by RDMA during execution. It achieves up to 2.96× speedup on the aggregation operator and 2.81× speedup on the join operator.