Aodhan Sweeney

by Aodhan Sweeney
2019 Unidata summer intern

This summer at Unidata I worked on expanding functionality for both the netCDF C++ library and the Python data access tool Siphon. Previously, the netCDF C++ library was lacking important functionality that was included in other netCDF libraries. Fortunately, adding this functionality is a straightforward process. I created function wrappers in the C++ library that would call previously made functions in the C library. This allows those working in a C++ framework to continue to use the netCDF libraries without sacrificing additional functionality.

This document defines the variant of the netcdf-c library API that can be used to read/write NCZarr dataset. Additionally, any special new flags or other parameter values are defined. It is expected that this document should be consistent with the NetCDF ZARR Data Model Specification [1].

This document defines the initial netcdf Zarr (NCZarr) data model to be implemented. As the Zarr version 3 specification progresses, this model will be extended to include new data types.

The Unidata NetCDF group is proposing to provide access to cloud storage (e.g. Amazon S3) by providing a mapping from a subset of the full netCDF Enhanced (aka netCDF-4) data model to one or more existing data models that already have mappings to key-value pair cloud storage systems.

The initial target is to map that subset of netCDF-4 to the Zarr data model [1]. As part of that effort, we intend to produce a set of related documents that provide a semi-formal definition of the following.

Unidata is in the process of developing a Zarr [] based variant of netcdf. As part of this effort, it was necessary to implement some support for chunking. Specifically, the problem to be solved was that of extracting a hyperslab of data from an n-dimensional variable (array in Zarr parlance) that has been divided into chunks (in the HDF5 sense). Each chunk is stored independently in the data storage -- Amazon S3, for example.

The algorithm takes a series of R slices of the form (first,stop,stride), where R is the rank of the variable. Note that a slice of the form (first, count, stride), as used by netcdf, is equivalent because stop = first + count*stride. These slices form a hyperslab.

The goal is to compute the set of chunks that intersect the hyperslab and to then extract the relevant data from that set of chunks to produce the hyperslab.