Indexing legacy data-sets for global access and processing in multi-cloud environments

Global data access and processing for multi-cloud applications proves challenging where there is a need for efficient access to large pre-existing, legacy data-sets. To address this problem, we created an indexing subsystem, allowing us to index and gather the metadata of legacy data-sets. The gathered information is later incorporated into a multi-cloud data management system to achieve global integration of legacy storage systems containing large data-sets. The solution is based on metadata management, organization and periodic monitoring of legacy data-sets, which makes possible scheme-less co-existence of a legacy storage system and a multi-cloud data management system for managing the same data collections. The approach has been initially evaluated in a multi-cloud deployment scenario. A large data collection consisting of legacy cultural data-sets was exposed to public and commercial infrastructure for data access and processing.