Parameterized deep reinforcement learning with hybrid action space for energy efficient data center networks

To ensure the delivery of high-performance and reliable services, data center networks (DCNs) are often over-provisioned for peak workload and traffic bursts. However, in real-world data centers, network traffic seldom reaches peak capacity of the network, resulting in significant energy waste. Traditional energy conservation approaches either suffer from high computational complexity and low solution quality, or their strategies cannot be dynamically adjusted to accommodate changes in data center network traffic. Deep reinforcement learning (DRL) provides an effective way to deal with these issues. However, most of the existing DRL-based schemes only consider either a continuous action space or a discrete action space, which greatly restricts the optimality of decisions. To solve these problems, this paper proposes a novel DRL-based DCN energy optimization framework, named SmartDCN. Specifically, SmartDCN consists of a traffic prediction module (TPM) and an energy optimization module (EOM). TPM incorporates an improved LSTM model JANET with an attention mechanism providing a high prediction accuracy, while EOM integrates our newly proposed parameterized DRL algorithm, named PAS-DQN, combining with the discrete-continuous hybrid action space. PAS-DQN implements a two-level control mechanism for the network, using TPM to predict future traffic in the data center as input. It is devoted to dynamically aggregating current traffic and makes tradeoffs between energy efficiency, performance, and robustness to optimize the network's power consumption by dynamically calculating the minimum required network subset and turning off the non-involved network devices to achieve power savings. Experimental results show that SmartDCN significantly outperforms the existing state-of-the-art schemes in terms of energy savings under various network conditions.