PIM: A visualization-oriented web application for monitoring and debugging of large-scale image processing studies

Advances in computer hard- and software have enabled the automated extraction of biomarkers from large scale imaging studies by means of image processing pipelines. For large cohort studies, ample storage- and computing resources are required: pipelines are typically executed in parallel on one or more High Performance Computing Clusters (HPC). As processing is distributed, it becomes more cumbersome to obtain detailed progress and status information of large-scale experiments. Especially in a research-oriented environment, where image processing pipelines are often in an experimental stage, debugging is a crucial part of the development process that relies heavily on a tight collaboration between pipeline developers and clinical researchers. Debugging a running pipeline is a challenging and time-consuming process for seasoned pipeline developers, and nearly impossible for clinical researchers, often involving parsing of complex logging systems and text files, and requires special knowledge of the HPC environment. In this paper, we present the Pipeline Inspection and Monitoring web application (PIM). The goal of PIM is to make it more straightforward and less time-consuming to inspect complex, long running image processing pipelines, irrespective of the level of technical expertise and the workflow engine. PIM provides an interactive, visualization-based web application to intuitively track progress, view pipeline structure and debug running image processing pipelines. The level of detail is fully customizable, supporting a wide variety of tasks (e.g. quick inspection and thorough debugging) and thereby facilitating both clinical researchers and pipeline developers in monitoring and debugging.