Daedalus: Self-Adaptive Horizontal Autoscaling for Resource Efficiency of Distributed Stream Processing Systems
arxiv(2024)
摘要
Distributed Stream Processing (DSP) systems are capable of processing large
streams of unbounded data, offering high throughput and low latencies. To
maintain a stable Quality of Service (QoS), these systems require a sufficient
allocation of resources. At the same time, over-provisioning can result in
wasted energy and high operating costs. Therefore, to maximize resource
utilization, autoscaling methods have been proposed that aim to efficiently
match the resource allocation with the incoming workload. However, determining
when and by how much to scale remains a significant challenge. Given the
long-running nature of DSP jobs, scaling actions need to be executed at
runtime, and to maintain a good QoS, they should be both accurate and
infrequent. To address the challenges of autoscaling, the concept of
self-adaptive systems is particularly fitting. These systems monitor themselves
and their environment, adapting to changes with minimal need for expert
involvement.
This paper introduces Daedalus, a self-adaptive manager for autoscaling in
DSP systems, which draws on the principles of self-adaption to address the
challenge of efficient autoscaling. Daedalus monitors a running DSP job and
builds performance models, aiming to predict the maximum processing capacity at
different scale-outs. When combined with time series forecasting to predict
future workloads, Daedalus proactively scales DSP jobs, optimizing for maximum
throughput and minimizing both latencies and resource usage. We conducted
experiments using Apache Flink and Kafka Streams to evaluate the performance of
Daedalus against two state-of-the-art approaches. Daedalus was able to achieve
comparable latencies while reducing resource usage by up to 71
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要