A stateful storage availability and entropy model to control storage distribution on grids

In spite of numerous works studying file replication strategies on distributed systems, data management policies remain mostly handled by manual operators or very basic algorithms on production grids. Among other causes, this situation is due to the lack of models taking job reliability into account. In this paper, we study file replication using new metrics to evaluate the reliability of distributed storage configurations. A stateful storage availability model is introduced to cope with the inability of the stateless model to account for the commonsense intuition that limiting the number of storage hosts involved in the execution of an application improves reliability. We describe the job success probability and the brittleness entropy, a metric describing the uncertainty of the job failure rate associated to a storage configuration. Results, obtained on synthetic data and on traces extracted from the European Grid Infrastructure, show that the stateful model is more accurate than the stateless on real data, and that it can describe the consequences of limiting the number of storage hosts on application reliability. These findings open the door to the design of new file replication strategies taking storage availability into account. Copyright © 2014 John Wiley & Sons, Ltd.