Machine Learning for Job-flow Online Scheduling and Resources Allocation in Distributed Computing

In this work we study a promising approach for efficient online scheduling of job-flows in high performance and distributed parallel computing. The majority of job-flow optimization approaches, including backfilling and microscheduling, require apriori knowledge of a full job queue to make the optimization decisions. In a more general scenario when user jobs are submitted individually, the resources selection and allocation should be performed immediately in the online mode. In this work we consider a neural network prototype model trained to perform online optimization decisions based on a known optimal solution. For this purpose, we designed MLAK algorithm which implements 0-1 knapsack problem based on the apriori unknown utility function. In a dedicated simulation experiments with different utility functions MLAK provides resources selection efficiency comparable to a classical greedy algorithm. The interdisciplinary nature of the considered problem and its solution (including elements from the areas of optimization, scheduling algorithms, machine learning and simulation) makes it a good study example in technical disciplines.