Lineage Checkpoint Approach for Long-lineage Problem in Apache Spark

In distributed data processing frameworks, data lineage is widely used to achieve fault-tolerance efficiently. However, as data analytics apps become more complex, long lineage incurs the performance and reliability issues. Existing data checkpoint solutions can alleviate the problem, however they bring new types of heavy inevitable overheads or lack fault-tolerance. To overcome the limitations, we propose the solution that checkpoints lineage graph instead of data itself, preserving the lineage information and reconciling the full lineage graph when needed. In evaluations, we show that our lineage checkpoint solution outperforms the data checkpoint solutions in terms of performance and fault-tolerance.