A Design of Parallel Content-Defined Chunking System Using Non-Hashing Algorithms on FPGA

Content-defined chunking is a common method in many applications such as data deduplication and data synchronization. In recent years, new CDC algorithms using non-hashing methods have been developed, and positive results have been obtained. However, most of the algorithms are developed for single-thread computation on microprocessors. After analyzing some popular CDC algorithms, we observed that the algorithms using the basic sliding window protocol are more feasible to process in parallel. In this work, we proposed a new parallel chunking method that aims for hardware implementation. Additionally, we used the PCI algorithm, which does not include hash functions, to implement a multi-thread chunking system on FPGA devices. By exploiting the strength of the FPGAs, our proposed design achieves not only high computational speed but also great scalability.