Data management in intelligent storage systems

Newly emerging data-intensive applications in business and scientific computing environments involve generating and processing a enormous amount of data. The requirements of storing and accessing data of such scale efficiently, securely and cost-effectively are pushing the edge of today's storage technology. New storage technologies like Internet SCSI (iSCSI) and Object-based Storage Device (OSD) have emerged to evolve storage architectures to the next generation of intelligent storage systems. The studies presented in this dissertation aim to address data management challenges in this new paradigm by exploring the intelligence of storage devices. We first investigate the data archiving issue in cluster computing environment. With fast improvements in CPU processing power and parallel I/O bandwidth of online storage devices, the archival storage I/O bandwidth becomes the system bottleneck. We propose a novel parallel Hierarchical Storage Management (HSM) architecture in Object-based Cluster File System (OCFS), which is highly scalable and fully automated. The fast and scalable migration bandwidth is achieved through initiating multiple parallel migration paths directly between OSDs. The automation of data migrations in storage hierarchy is realized through collaboration of distributed HSM instances embedded in OSDs. We have addressed the issues including coordinating the parallel migration paths, guaranteeing consistency of data status, recovering from component failures and object placement. In the second part, we investigate the features and design of OSD based on tape devices, which are essential components in OCFS with storage hierarchy. We propose extensions of the object storage interface to address the need of object archiving applications as well as the sequential-access characteristics of tape devices. We further address a set of self-management issues of the object tape libraries. Finally, we investigate the data sharing issue in remote-accessible storage devices. We study how to provide consistent data sharing for file systems that use remote iSCSI targets. We propose extensions of the SCSI command set to make iSCSI target be aware of file system semantics. Based on this, we design a suite of mechanisms to guarantee metadata, regular data and cache consistency under concurrent sharing and client caching.