A Distributed Service Architecture for end-to-end Data Intensive Analysis

Grid computing has become a popular way of providing high performance for data intensive scientific applications. Many interesting and challenging problems related to data intensive computing have been solved recently using various grid services. However, reliable and scalable software architecture for solving general-purpose distributed data intensive problems is missing. We developed a software architecture that combines existing grid services with our state-of-the-art grid scheduler, Sphinx. We deployed our prototype across the USCMS Grid3 (Grid3). It is in the primary stages of exhibiting interactive remote data access, demonstrating interactive workflow generation and collaborative data analysis using virtual data and data provenance, as well as showing non-trivial examples of policy based scheduling of requests in a resource constrained grid environment. Here we focus on the design of infrastructure that handles tasks from the generation of abstract workflows to analysis of results. We present our experiments in scheduling various workflows using different algorithms in Sphinx.