Multi-Objective Optimization of Consumer Group Autoscaling in Message Broker Systems
CoRR(2024)
摘要
Message brokers often mediate communication between data producers and
consumers by adding variable-sized messages to ordered distributed queues. Our
goal is to determine the number of consumers and consumer-partition assignments
needed to ensure that the rate of data consumption keeps up with the rate of
data production. We model the problem as a variable item size bin packing
problem. As the rate of production varies, new consumer-partition assignments
are computed, which may require rebalancing a partition from one consumer to
another. While rebalancing a queue, the data being produced into the queue is
not read leading to additional latency costs. As such, we focus on the
multi-objective optimization cost of minimizing both the number of consumers
and queue migrations. We present a variety of algorithms and compare them to
established bin packing heuristics for this application. Comparing our proposed
consumer group assignment strategy with Kafka's, a commonly employed strategy,
our strategy presents a 90th percentile latency of 4.52s compared to Kafka's
217s with both using the same amount of consumers. Kafka's assignment strategy
only improved the consumer group's performance with regards to latency with
configurations that used at least 60
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要