Run-Length Base-Delta Encoding For High-Speed Compression

In modern supercomputers, nodes are connected by networking hardware capable of up to 40 Gb/s. Data compression could allow for even higher effective bandwidth. However, data compression for such systems requires a unique tradeoff between the compression rate delivered by the compressing scheme and the speed of compression/decompression. While traditional software compression techniques may deliver high compression rates, they cannot maintain the high compression/decompression throughput that is needed. We present Run-Length Base-Delta (RLBD) encoding, a software compression format and algorithm that delivers a high-speed compression/decompression that is suitable for data transfers at up to 40GbE. RLBD can be implemented in CPUs or in parallel accelerator devices to improve network throughput by up to 57% while transmitting data from real datasets.