Cider: A Case For Block Level Variable Redundancy On A Distributed Flash Array

With the increase in data volumes, it is prudent to classify data depending on its criticality. One might prefer a cheap storage for a year old system logs but a highly fault tolerant storage for personal photos. The existing solutions include storing these two sets of data in two different systems or choosing a system with a fault tolerance level required by the most critical data. This means a higher storage footprint for the lesser critical data. In today's petabyte scale data centers, this will result in a significant increase in the overall storage footprint making the system unattractive. In addition, data storage solutions designed for traditional data centers have to be re-engineered to work at cloud scale. For instance, reliable storage using traditional RAID like systems are inflexible and do not provide a mechanism to easily change the level of redundancy once the system is set up. As a result, changes to the reliability requirements of data over time cannot be serviced efficiently.To address these problems, we propose Cider, which aims at providing an extremely flexible, reliable, and distributed block store using erasure codes. Cider takes a new approach to reduce the storage overhead by offering a variable degree of fault tolerance at a granularity of a single block. We achieve this by using a thin block translation layer and a block level metadata system. Cider can be readily used by many high performance and enterprise applications which require a block level interface. Through a case study of a novel flash based clustered storage system, we make a strong case for the adoption of Cider in future systems. We discuss various design trade-offs and implementation challenges associated with designing our system. Lastly, we discuss the preliminary implementation of our system and results from our initial experiments.