The Open Datacube Federation

<p>Datacubes are acknowledged as a cornerstone for analysis-ready data as they allow more intuitive, human-centric services. In the GAIA-X EO Expert Group, a subgroup of the GAIA-X Geoinformation Working Group, datacube federations are one of the use cases investigated, specifically: the EarthServer initiative which has set out to materialize the vision of a single integrated, homogenized, location-transparent datacube pool. It bridges a seeming contradiction: a decentralized approach of independent data providers - with heterogeneous offerings, paid as well as free - versus a single, common pool of datacubes where users do not need to know where data sit inorder to access, analyse, mix, and match them. In analogy to the term &#8220;server-less&#8221; such a federation might be called &#8220;datacenter-less&#8221; as users do not need to know the concrete data location any longer. <br /><br />Among the service parameters achieved are: <br />- zero-coding, human-oriented space/time analytics;<br />- datacubes uniformly offered through the OGC/ISO/INSPIRE datacube service standards, enabling a wide range of 3rd-party clients with seamless access, such as python xarray, numpy arrays, OpenLayers, Leaflet, NASA WebWorldWind, Microsoft Cesium, QGIS, ArcGIS, and others more;<br />- Server-side extensibility for integrating any 3rd-party code into the service orchestration, such as ML models;<br />- transparent distributed data fusion;<br />- no single point of failure;<br />- seamless offering of both free and paid data and services;<br />- administrator-less continuous growth of datacubes over time, allowing data centers to join the service without assigning dedicated staff resources;<br />- Fine-grain access control, down to single-pixel granularity. <br /><br />The large, continuously growing EarthServer federation (the latest member joining was the Taiwan National Supercomputing Center) is boosted by rasdaman, the pioneer datacube engine whose query language in 2019 has been adopted as the SQL datacube extension. Being the official INSPIRE Good Practice rasdaman effectively integrates Copernicus and INSPIRE seamlessly, paving the way for mapping agencies joining the federation. The aggressive growth of the EarthServer federation is ongoing; a line-up of datacenters has expressed interest, and the charter for governance is being finalized. <br /><br />As of today, EarthServer offers a critical mass of 140+ of Petabytes of multi-dimensional raster data, including 2D DEMs, 3-D satellite image timeseries, and 4-D atmospheric data. Members include several DIAS European Copernicus archives, leading supercomputing research centers, as well as a series of specialized services offering high-level marine, land use, and atmospheric products. All data are accessible with zero coding, in particular: without the need to know python, and strictly standards compliant. Virtual Coverages allow users to see single datacubes even where the underlying data are heterogeneous. <br /><br />In our talk we present the federation rationales and opportunities and show a broad range of real-life distributed data fusion examples.<br /><br /></p>