A case for dynamic memory partitioning in data centers.

ABSTRACTLeveraging distributed main memory is becoming an increasingly popular approach to speed up large-scale data-intensive cluster applications. However, despite the growing number of possible performance benefits, recent studies indicate that the static resource partitioning among different applications and users in those clusters often leads to severe memory fragmentation, rendering almost half of the available memory resources unusable. This paper therefore proposes to extend the static memory partitioning of current cluster resource managers by a more dynamic scheme which continues to ensure a fair resource distribution among the tenants but allows individual applications to claim spare main memory on a temporary basis. We show that our new approach is a natural fit for many use cases in the big data domain and can significantly improve the memory utilization and processing efficiency.