An Intelligent Scheduling System for Large-Scale Online Judging

Online judge (OJ) systems have been widely used for programming skill evaluation in various fields, including programming education, programming competition and talent recruitment. Existing OJ systems put the codes into a judge queue according to the order of user submission, and use the judge server to evaluate the correctness of the codes in turn. With the surge in the number of code submissions, this scheduling method causes the rapid increase of average response time for judge requests, resulting in a decline in user experience. To alleviate the problem, we develop an intelligent scheduling system, which consists of two modules. In the first module, we employ a deep representation learning model to predict the running time of the codes in the judge queue; in the second module, the judge queue is divided into fixed-size windows. The codes in each window are sorted according to their predicted running time in ascending order, and are scheduled to the judge server using the shortest job first algorithm. The experimental results show that, 1) the constructed prediction model predicts the running time of the codes accurately; 2) compared with the scheduling algorithm of existing OJ systems, the proposed scheduling algorithm can effectively reduce the average response time for large-scale online judging. Furthermore, by varying the code running time distribution and window size in the judge queue, we demonstrate the performance improvements of the proposed intelligent scheduling system under different settings, compared with the existing systems.