A Comprehensive Benchmarking Analysis of Fault Recovery in Stream Processing Frameworks
CoRR(2024)
摘要
Nowadays, several software systems rely on stream processing architectures to
deliver scalable performance and handle large volumes of data in near real
time. Stream processing frameworks facilitate scalable computing by
distributing the application's execution across multiple machines. Despite
performance being extensively studied, the measurement of fault tolerance-a key
and most appealing feature offered by stream processing frameworks-has still
not been measured properly with updated and comprehensive testbeds. Moreover,
the impact that fault recovery can have on performance is mostly ignored. This
paper provides a comprehensive analysis of fault recovery performance,
stability, and recovery time in a cloud-native environment with modern
open-source frameworks, namely Flink, Kafka Streams, and Spark Structured
Streaming. Our benchmarking analysis is inspired by chaos engineering to inject
failures. Generally, our results indicate that much has changed compared to
previous studies on fault recovery in distributed stream processing. In
particular, the results indicate that Flink can be the fastest and stablest
under failures. Moreover, Kafka Streams shows performance instabilities after
failures, which is due to its current repartitioning strategy that can be
suboptimal in terms of load balancing. Spark Structured Streaming shows
suitable fault recovery performance and stability, but with higher event
latency. Our study intends to (i) help industry practitioners in choosing the
most suitable stream processing framework for efficient and reliable executions
of data-intensive applications; (ii) support researchers in applying and
extending our research method as well as our benchmark; (iii) identify,
prevent, and assist in solving potential issues in production deployments.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要