High-level Stream Processing: A Complementary Analysis of Fault Recovery
arxiv(2024)
摘要
Parallel computing is very important to accelerate the performance of
software systems. Additionally, considering that a recurring challenge is to
process high data volumes continuously, stream processing emerged as a paradigm
and software architectural style. Several software systems rely on stream
processing to deliver scalable performance, whereas open-source frameworks
provide coding abstraction and high-level parallel computing. Although stream
processing's performance is being extensively studied, the measurement of fault
tolerance–a key abstraction offered by stream processing frameworks–has still
not been adequately measured with comprehensive testbeds. In this work, we
extend the previous fault recovery measurements with an exploratory analysis of
the configuration space, additional experimental measurements, and analysis of
improvement opportunities. We focus on robust deployment setups inspired by
requirements for near real-time analytics of a large cloud observability
platform. The results indicate significant potential for improving fault
recovery and performance. However, these improvements entail grappling with
configuration complexities, particularly in identifying and selecting the
configurations to be fine-tuned and determining the appropriate values for
them. Therefore, new abstractions for transparent configuration tuning are also
needed for large-scale industry setups. We believe that more software
engineering efforts are needed to provide insights into potential abstractions
and how to achieve them. The stream processing community and industry
practitioners could also benefit from more interactions with the high-level
parallel programming community, whose expertise and insights on making parallel
programming more productive and efficient could be extended.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要