Deduplication: Concepts and Techniques

This document’s goal is to present a survey of the general deduplication design space. Deduplication is a broad concept, and deduplication systems can be implemented using a variety of techniques and design choices. For each core design element, the document presents the high-level idea and a set of considerations that influence an appropriate choice for that design. The document also presents a few representative systems as case studies, hopefully giving a sense for practical parameter choices and their implications. If there are parts of the text that are unclear, please let me know so that I can improve them! I am not aware of any good surveys/introductory texts on deduplication, so this is my attempt to create one for us to use in this class.