A framework for data management and transfer in grid environments

The main obstacles to grid file management come from the fact that grid file resources are typically stored in heterogeneous and distributed environment and accessed through various protocols. In this paper, we propose a grid file management system called Vega [1][2] Hotfile2 for data-intensive application in widely distributed systems and grid environments. Widely distributed and heterogeneous storage resource can converge into a single uniform storage resource using virtualization techniques in Hotfile2. It is a grid middleware provides a uniform file resources view and a high level interface to access widely distributed and heterogeneous data resources. Presented here are two services that we believe are fundamental to any grid file management system: high-performance, secure transport and metadata management. In this paper, we discuss many aspects of our design and implementation in Hotfile2, present the file system interface designed to support distributed applications such as batch job. Experiments are done to illustrate its performance, extensibility, and various other features