DAG-Aware Optimization for Geo-Distributed Data Analytics

Geo-distributed data analytics has been proposed to analyze geographically distributed data. Existing studies have achieved significant reductions in execution time and data transfer cost ($) of data analytics jobs by optimizing task placement. Given a directed acyclic graph (DAG)-style job, however, they mainly optimize each stage independently, and they tend to distribute tasks and intermediate data across all locations, potentially inflating execution time and data transfer cost of descendent stages and the whole job. In this paper, we propose a DAG-aware approach to minimize job data transfer costs while guaranteeing job execution time. Specifically, we design a two-phase static/runtime algorithm that is both lightweight and adaptive to dynamics. The static phase estimates the optimal placement of all stages in the job, minimizing the job data transfer cost. Then for each stage ready to be executed, the runtime phase re-optimizes its task placement based on the static task placement of child stages and runtime information. It minimizes the stage data transfer cost while incorporating the stage execution time with a simple control knob. Overall, our approach properly aggregates early-stage tasks to fewer data centers, thereby reducing subsequent stages and whole job data transfer cost and execution time. We implement our approach in Spark and evaluate it across geo-distributed datacenters. Our approach reduces application data transfer cost by up to 91% without increasing job execution time compared to existing baselines.