The Library for Hadoop Deflate Compression Based on FPGA Accelerator with Load Balance

Hadoop application will produce lots of intermediate results in the map/reduce process that requires disk I/O and network transmission. By compressing the large-scale data of intermediate result, it will greatly improve disk access efficiently and reduce program run time. Hardware-accelerated solutions have become more desirable. This paper design a multi-FPGA compression accelerator on the Hadoop platform, and the system performance analysis compared with a software-only solution that mainly uses CPU to processing. The testing programs are zpipe, TestDFSIO and Terasort. In contrast with the software-only solution. The max speedup of zpipe is 6.55X (single FPGA) and 10.24X (dual FPGA), the max speedup of TestDFSIO is 6.28X (single FPGA) and 6.28X (dual FPGA), and the max speedup of Terasort application is up to 3.25X(single FPGA) and 3.35X(dual FPGA).