A Multi-agent Parallel Approach to Analyzing Large Climate Data Sets

Despite various cloud technologies that have parallelized and scaled up big data analysis, they target data mostly in texts which are easy to partition and thus easy to map over a cluster system. Therefore, their parallelization do not necessarily cover scientific structured data such as NetCDF or need additional, user-provided tools to convert the original data into specific formats. To facilitate user-intuitive parallelization of such scientific data analysis, this paper presents an agent-based approach that instantiates distributed arrays over a cluster system, maintains structured scientific data in these arrays, deploys many mobile agents over the arrays to perform computational actions on data, and collects necessary results. To demonstrate the practicability of our agent-based approach, we focused on climate change research and implemented a web-interfaced climate analysis, using the MASS (multi-agent spatial simulation) library. In this paper, we show practical advantages of, performance improvements by, and challenges for our agent-based approach in structured data analysis.