Cross-contextual Sequential Optimization via Deep Reinforcement Learning for Algorithmic Trading

High-frequency algorithmic trading has consistently attracted attention in both academic and industrial fields, which is formally modeled as a near real-time sequential decision problem. DRL methods are treated as a promising direction compared with the traditional approaches, as they have shown great potential in chasing maximum accumulative return. However, the financial data gathered from volatile market change rapidly, which makes it dramatically difficult to grasp crucial factors for effective decision-making. Existing works mainly focus on capturing temporal relations while ignoring deriving essential factors across features. Therefore, we propose a DRL-based cross-contextual sequential optimization (CCSO) method for algorithmic trading. In particular, we employ a convolution module in the first stage to derive latent factors via inter-sequence aggregation and apply a well-designed self-attention module in the second stage to capture market dynamics by aggregating temporal intra-sequence details. With the two-stage extractor as encoder and a RNN-based decision-maker as decoder, an Encoder-Decoder module is established as the policy network to conduct potent feature analysis and suggest action plans. Then, we design a dynamic programming based learning method to address the challenge of complex network updates in reinforcement learning, leading to considerable enhancement in learning stability and efficiency. To the best of our knowledge, this is the first work that solves the sequential optimization problem by joint representation of trading data across time and features in the DRL framework. Extensive experiments demonstrate the superior performance of our method compared to other state-of-the-art algorithmic trading approaches in various widely-used metrics.