Development Of Complex Scientific Workflows: Towards End-To-End Workflows

The analysis of water planning options on environmental assets relies on combining mathematical models from several disciplines. The growing complexity of these modelling tasks increases the potential for mistakes and misinforming stakeholders and the public. Through better capture of provenance information (audit trails), scientific workflow tools improve the transparency of model interactions, which increases our confidence in the modelling results. However, scientific workflow tools can be complex to use and increase software development costs. Consequently, they have not had widespread adoption.This paper examines progress toward end-to-end scientific workflows. These end-to-end workflows link heterogeneous data sources through to reports that evaluate ecosystem function. The aim of end-to-end workflows is to provide a system that can evolve as understanding progresses, data services come online and reporting requirements change.The case study for this investigation is Little Rushy Swamp. Located in the Barmah forest near Echuca, Little Rushy Swamp supports a range of bird life. Over the past century, human regulation of the flows of the River Murray has changed the timing and frequency of flooding, which has caused the deterioration in Red Gum (Eucalyptus camaldulensis) health and wetland habitats. In recent years, the Australian Government has bought water licenses with the aim of improving the health of riparian ecosystems. The Government has used their water licenses to provide significant environmental flooding for the Barmah Forest. This case study shows how to link a model that characterises wetland hydrology in a way that supports ongoing reporting requirements that may be necessary for monitoring or management.The workflow was designed using tools from the Hydrologists Workbench, which aims to ease the development of complex automated workflows in Trident. In the workflow, we use a simple water balance model to understand how the water level of Little Rushy Swamp varies under different climatic periods. The water balance model has daily inputs. The inputs to the model include data from text files, web feature services and outputs from other simple numerical models. The results are analysed using R-based statistics and ArcGIS-based geoprocessing. The workflow exports resulting images to Microsoft SharePoint using its web service interface. Using links, the images are incorporated into a Microsoft Word document. The Word document updates when the images on SharePoint update, providing an end-to-end workflow.We find that the end-to-end workflow for Little Rushy Swamp addresses a number of challenges that exist in the integrated environmental modelling space. In particular, we establish that the workflow provides a valuable tool for incorporating new or revised datasets and methods. However, the benefits of such workflows are limited by the availability of web service data feeds that use consistent data formats. Further work should be directed towards handling of uncertainty by workflow systems.