N-Storm - Efficient Thread-Level Task Migration in Apache Storm.

Apache Storm is a widely used stream processing system, but it only supports offline task scheduling. Recently, some online task scheduling approaches were proposed to offer task migrations in Storm at runtime. However, most of them have to cost more than 10 seconds during a task migration, because they employed a Worker-level scheme to stop or start Workers (implemented as processes in Storm). When an Executor within a Worker needs migrating, Storm has to kill the Worker, leading to the stop and restart of all Executors in the Worker. This process-level scheme introduces unnecessary stop and restart of Executors and Workers, resulting in poor performance of task migrations. Aiming to solve this problem, we propose N-Storm (Non-stop Storm). Instead of using the process-level migration scheme, N-Storm employs a thread-level scheme for task migrations. Particularly, we add a key/value store on each Worker node to make Workers be aware of the changes of the task schedule, so that the Workers on a Worker node can manage their Executors at runtime, i.e., killing existing Executors or starting new Executors. With this mechanism, we can avoid the unnecessary stop of Executors and Workers during a task migration, and thus improve the performance of task migrations. We further propose two optimizations to make N-Storm efficient for multiple tasks migrations. Our experimental results on a real Storm cluster show that N-Storm is able to reduce the stop time, increase the system throughput, and save significant migration time, compared with Storm and other approaches.