Magma: A high data density storage engine used in Couchbase.

We present Magma, a write-optimized high data density key-value storage engine used in the Couchbase NoSQL distributed document database. Today's write-heavy data-intensive applications like ad-serving, internet-of-things, messaging, and online gaming, generate massive amounts of data. As a result, the requirement for storing and retrieving large volumes of data has grown rapidly. Distributed databases that can scale out horizontally by adding more nodes can be used to serve the requirements of these internetscale applications. To maintain a reasonable cost of ownership, we need to improve storage e.ciency in handling large data volumes per node, such that we don't have to rely on adding more nodes. Our current generation storage engine, Couchstore is based on a log-structured append-only copy-on-write B+Tree architecture. To make substantial improvements to support higher data density and write throughput, we needed a storage engine architecture that lowers write ampli.cation and avoids compaction operations that rewrite the whole database.les periodically. We introduce Magma, a hybrid key-value storage engine that combines LSM Trees and a segmented log approach from logstructured.le systems. We present a novel approach to performing garbage collection of stale document versions avoiding index lookup during log segment compaction. This is the key to achieving storage e.ciency for Magma and eliminates the need for random I/Os during compaction. Magma o.ers signi.cantly lower write ampli.cation, scalable incremental compaction, and lower space ampli.cation while not regressing the read ampli.cation. Through the e.ciency improvements, we improved the single machine data density supported by the Couchbase Server by 3.3x and lowered the memory requirement by 10x, thereby reducing the total cost of ownership up to 10x. Our evaluation results show that Magma outperforms Couchstore and RocksDB in write-heavy workloads.