Combating Workflow Failures with Integrity-based Checkpoints and Blockchain 3 rd Mats Rynge 5 th

Workflow management systems are subject to failures , including: processor, network congestion, and machine reboot. Various fault tolerance techniques have been proposed to address these failures. Data integrity errors also cause workflows to fail, but little or no attention has been given to integrity faults. The Scientific Workflow Integrity with Pegasas (SWIP) project has shown data integrity errors do occur in the wild. These errors occur when transferring and storing experiment data. The inability of today's validation mechanisms such as TCP check-sums and Layer 2 checksums, motivated the SWIP project to add an extra layer of application layer data integrity verification using cryptographic hashes. Currently, the SWIP project takes a checkpoint all approach for integrity data; moving all integrity data for a task to stable storage. In this paper, we characterize nodes in workflow graphs based on the graph structure and propose several integrity-based checkpointing strategies. These strategies use a node's properties to determine which nodes to checkpoint. When failures occur the proposed integrity-based checkpointing strategies allow us to validate the integrity of data of preceding workflow tasks and re-use data during workflow retries. This paper focuses on characterizing workflow graphs to identify key nodes, thereby reducing the overheads of workflow retries. We also explore and evaluate the blockchain technology to securely preserve the integrity meta-data.