A Highly Available Distributed Self-Scheduler For Exascale Computing

A hierarchical master-worker model is thought to be a promising programming paradigm for exascale-level high performance computers. However, "fault resiliency" is one of the most important issues for exascale computing because the Mean Time Between Failure (MTBF) is expected to be short. We propose a fault resilient middleware suite for exascale computing environments. In this paper, we design a highly available distributed self-scheduler as a resource management system for the proposed middleware suite. The proposed distributed self-scheduler consists of multiple processes in order to achieve scalability, fault resiliency, and persistency. We also develop a prototype system of the middleware, using Apache ZooKeeper and Apache Cassandra. Experiments using the developed prototype system show that the proposed distributed self-scheduler achieves the desired fault resiliency for an application program developed using the middleware, and that the scheduler itself is also fault resilient. We also confirmed that the overheads caused by distributed processing can be reduced, and the scheduler can be scalable.