The Unidata NetCDF group is proposing to provide access to cloud storage (e.g. Amazon S3) by providing a mapping from a subset of the full netCDF Enhanced (aka netCDF-4) data model to one or more existing data models that already have mappings to key-value pair cloud storage systems.
The initial target is to map that subset of netCDF-4 to the Zarr data model [1]. As part of that effort, we intend to produce a set of related documents that provide a semi-formal definition of the following.
- A description of the initial NCZarr data model.
- A description of the subset of the netCDF API that conforms to the NCZARR data model. This interface will be the basis for programatically reading and writing cloud data via the netcdf-c library.
- A mapping of the NCZarr data model to some variant of the Zarr storage representation, This representation is a combination of a mapping to Json plus a mapping to an abstract key-value pair interface.
- The internal architecture of the cloud support in the netcdf-c library.
- Any other documents required in support of the preceding documents (the chunking algorithm documents, for example).
The term "semi-formal" is used because rather than provide complete mathematical or operational semantics, prose text will be used to describe the context-sensitive features of the model. A complete formalization in order to produce an operationally defined specification is a possible future activity.
References
[1] Zarr storage specification version 2 (https://zarr.readthedocs.io/en/stable/spec/v2.html)
Copyright
Copyright 2018, UCAR/Unidata
See netcdf/COPYRIGHT file for copying and redistribution conditions.
Point of Contact
Author: Dennis Heimbigner
Email: dmh at ucar dot edu
Initial Version: 11/28/2018
Last Revised: 7/2/2019
