An Intelligent Data Placement Strategy for Hierarchical Storage Systems

With the arrival of big data era, hierarchical storage systems are commonly used to optimize the I/O performance. While data placement strategies have significant influence on the I/O performance, the traditional one-layout-fits-all strategy fails to make optimal decisions and could not utilize hierarchical storage architecture efficiently. Some strategies based on user hints cannot meet the demand of data explosion. Although recent efforts propose several adaptive data placement techniques, fitting diverse scientific workflows and complex system characteristics are still challenging. This paper presents WorkflowRL, an intelligent data placement engine that utilizes reinforcement learning to manage data across multi-layer hierarchical storage system. WorkflowRL extracts factors affecting the I/O performance including the workflow characteristics and system information for feature learning. Then, it interacts with the heterogeneous storage system environment to make optimal data placement decisions. Extensive experiments have been conducted and the results have demonstrated that our data placement strategy can significantly improve the I/O performance with diverse scientific workflows.