Online Encoding for Erasure-Coded Distributed Storage Systems

Many large-scale distributed storage systems deploy erasure coding to protect data from frequent server failures for cost reason. In most of these systems, newly inserted data is first replicated across different storage servers and then migrated to erasure coded. Although this offline encoding manner can improve data access before data is erasure coded for some storage systems, it helps little and wastes many critical network and disk resources for many other storage systems. In this study, we propose an online encoding method, which encodes data as soon as it is inserted into the system. By eliminating the migration process, our online encoding can significantly reduce network transfer and data read; by caching the intermediate parity blocks into memory, our online encoding also significantly reduce data write. Analysis show that our online encoding can reduce data transfer by more than 25%, reduce data write by 57% at least and eliminate all data read, compared to traditional offline encoding. Experiments on a real cluster show online encoding reduces insert time by 20%-24%.