Pipeline Group Optimization on Disaggregated Systems

While hardware disaggregation is considered the "next big thing" providing unique opportunities for database systems, the pipeline-based execution model is state-of-the-art in modern query engines on monolithic systems. Within this paper, we propose a lightweight way of adapting this pipeline-based model to disaggregated memory systems to soften the inherent overhead induced by arbitrary memory accesses. Instead of executing pipelines in strict isolation including a pipeline-local data transfer, we group pipelines with similar data access characteristics of concurrently running queries into pipeline groups . Each such pipeline group is then executed sep-arately, but shared data across pipelines within each group is only transferred once from memory resources to compute resources and potentially re-used multiple times. This method dramatically re-duces redundant data transfers and – in combination with a suitable caching strategy as well as a fast communication layer – increases the performance significantly in comparison to traditional pipeline-based execution of multiple queries.