Reducing Average Job Completion Time for DAG-style Jobs by Adding Idle Slots.

Sizes of data processing jobs in cloud clusters have been growing rapidly in the big data era. It is critical to execute those jobs efficiently. The average job completion time (JCT) is a widely used metric to measure executing efficiency. JCT refers to the length of the time interval between a job's arrival to its completion. Typically, a data processing job contains multiple stages with complex precedence constraints. Carefully scheduling the processing sequence of stages within a job may significantly reduce its JCT. Our objective is to minimize the average JCT for online arrival jobs. The computation graphs of those jobs are usually directed acyclic graphs (DAGs). It makes the scheduling problem challenging. Recent works have shown that reinforcement learning (RL) agents can adaptively adjust the scheduling policies by dynamically assigning priorities for job stages. However, we notice that other factors besides stage priories may impact the JCT significantly. In particular, we observe that inserting idle slots before large jobs may reduce the waiting time of small jobs that arrive slightly later and reduce the average JCT. We analyze the benefits of inserting idle time for simple cases theoretically and show the condition in which idle slots should be inserted for two adjacent jobs. In addition, we adapt the RL-based scheduler by integrating the observation. Experiment results on both real-world and synthetic datasets show the efficiency of our scheduler. Also, a perturbation-based method is applied to demonstrate the contribution of each proposed feature.