Hi, Ward!
I'm trying to convert a pure Zarr to GeoTIFF via GDAL and
HDF5 via NCO using the new netCDF library.
However, I have some trouble in following your NUG:
"NCZarr versus Pure Zarr.
The NCZARR format extends the pure Zarr format by adding extra objects such
as _.nczarr_ and _.ncvar_. It is possible to suppress the use of these
extensions so that the netcdf library can read and write a pure zarr
formatted file. This is controlled by using mode=nczarr,zarr combination.
The primary effects of using pure zarr are described in the Translation
Section."
I tried a few different modes but none worked for reading Zarr file:
~/src/netcdf-c-4.8.0/ncdump$ ./ncdump -h
"file:///efs/power_901_constants.zarr/#mode=zarr,file"
/home/ubuntu/src/netcdf-c-4.8.0/ncdump/.libs/ncdump:
file:///efs/power_901_constants.zarr/#mode\
=zarr,file: NetCDF: internal library error; Please contact Unidata support
~/src/netcdf-c-4.8.0/ncdump$ ./ncdump -h
"file:///efs/power_901_constants.zarr/#mode=nczarr,file"
ncdump: zclose.c:223: zclose_type: Assertion `type &&
type->format_type_info != NULL' failed.
Aborted (core dumped)
Am I correct in using the "mode=zarr,file" for pure Zarr?
By the way, ncdump works fine on sample test nczarr files.
Regards,
- Joe
---
Deploy XaaS quickly using HDF in Distributed Cloud.
On Tue, Mar 30, 2021 at 5:39 PM Ward Fisher <wfisher@xxxxxxxx> wrote:
> Hello all,
>
> The Unidata netCDF team is happy to announce the availability of netCDF
> 4.8.0, now with Zarr support (nczarr)!
>
> This release extends the netcdf-c library to provide access to cloud
> storage (e.g. Amazon S3 by providing a mapping from a subset of the full
> netCDF Enhanced (aka netCDF–4) data model to a variant of the Zarr data
> model. The NetCDF variant is called NCZarr. It is possible for other
> implementations, such as Python Zarr, to read this variant. This
> implementation can optionally read and write using the pure version 2 Zarr
> storage format. It also can optionally support the XArray _ARRAY_DIMENSIONS
> extension to Zarr.
>
> In addition to this new functionality, the release notes appended to the
> end of this announcement detail some of the other changes, improvements and
> bug fixes included in this release.
>
> Please see the following links for more information:
>
> - Github release page:
> https://github.com/Unidata/netcdf-c/releases/tag/v4.8.0
> - v4.8.0 Documentation:
> https://www.unidata.ucar.edu/software/netcdf/docs
> - NCZarr Documentation:
>
> https://www.unidata.ucar.edu/software/netcdf/documentation/NUG/nczarr_head.html
>
> Kudos to Dennis Heimbigner for the technical implementation of nczarr, and
> our thanks to the members of the Zarr working group and our community who
> have collaborated with us so patiently on this!
>
> Our next priority is to get caught up on the backlog of PRs and issues
> blocking the next releases of netCDF-Fortran and netCDF-CXX4.
>
> Thank you all very much, have a great day!
>
> -Ward
>
> Ward Fisher
> wfisher@xxxxxxxx
> UCAR/Unidata - Software Engineer
> NetCDF Team Lead
> Full Release Notes4.8.0 - March 30, 2021
>
> - [Enhancement] Bump the NC_DISPATCH_VERSION from 2 to 3, and as a
> side effect, unify the definition of NC_DISPATCH_VERSION so it only needs
> to be defined in CMakeLists.txt and configure.ac. See Github #1945
> <https://github.com/Unidata/netcdf-c/pull/1945> for more information.
> - [Enhancement] Provide better cross platform path name management.
> This converts paths for various platforms (e.g. Windows, MSYS, etc.) so
> that they are in the proper format for the executing platform. See Github
> #1958 <https://github.com/Unidata/netcdf-c/pull/1958> for more
> information.
> - [Bug Fixes] The nccopy program was treating -d0 as turning deflation
> on rather than interpreting it as “turn off deflation”. See Github
> #1944 <https://github.com/Unidata/netcdf-c/pull/1944> for more
> information.
> - [Enhancement] Add support for storing NCZarr data in zip files. See
> Github
> #1942 <https://github.com/Unidata/netcdf-c/pull/1942> for more
> information.
> - [Bug Fixes] Make fillmismatch the default for DAP2 and DAP4; too
> many servers ignore this requirement.
> - [Bug Fixes] Fix some memory leaks in NCZarr, fix a bug with long
> strides in NCZarr. See Github #1913
> <https://github.com/Unidata/netcdf-c/pull/1913> for more information.
> - [Enhancement] Add some optimizations to NCZarr, dosome cleanup of
> code cruft, add some NCZarr test cases, add a performance test to NCZarr.
> See Github #1908 <https://github.com/Unidata/netcdf-c/pull/1908> for
> more information.
> - [Bug Fix] Implement a better chunk cache system for NCZarr. The
> cache now uses extendible hashing plus a linked list for provide a
> combination of expandibility, fast access, and LRU behavior. See Github
> #1887 <https://github.com/Unidata/netcdf-c/pull/1887> for more
> information.
> - [Enhancement] Provide .rc fields for S3 authentication:
> HTTP.S3.ACCESSID and HTTP.S3.SECRETKEY.
> - [Enhancement] Give the client control over what parts of a DAP2 URL
> are URL encoded (i.e. %xx). This is to support the different decoding rules
> that servers apply to incoming URLS. See Github #1884
> <https://github.com/Unidata/netcdf-c/pull/1844> for more information.
> - [Bug Fix] Fix incorrect time offsets from ncdump -t, in some cases
> when the time units attribute contains both a *non-zero* time-of-day,
> and a time zone suffix containing the letter “T”, such as “UTC”. See Github
> #1866 <https://github.com/Unidata/netcdf-c/pull/1866> for more
> information.
> - [Bug Fix] Cleanup the NCZarr S3 build options. See Github #1869
> <https://github.com/Unidata/netcdf-c/pull/1869> for more information.
> - [Bug Fix] Support aligned access for selected ARM processors. See Github
> #1871 <https://github.com/Unidata/netcdf-c/pull/1871> for more
> information.
> - [Documentation] Migrated the documents in the NUG/ directory to the
> dedicated NUG repository found at https://github.com/Unidata/netcdf
> - [Bug Fix] Revert the internal filter code to simplify it. From the
> user’s point of view, the only visible change should be that (1) the
> functions that convert text to filter specs have had their signature
> reverted and renamed and have been moved to netcdf_aux.h, and (2) Some
> filter API functions now return NC_ENOFILTER when inquiry is made about
> some filter. Internally, the dispatch table has been modified to get rid of
> the complex structures.
> - [Bug Fix] If the HDF5 byte-range Virtual File Driver is available
> )HDf5 1.10.6 or later) then use it because it has better performance than
> the one currently built into the netcdf library.
> - [Bug Fix] Fixed byte-range support with cURL > 7.69. See [
> https://github.com/Unidata/netcdf-c/pull/1798].
> - [Enhancement] Added new test for using compression with parallel
> I/O: nc_test4/tst_h_par_compress.c. See [
> https://github.com/Unidata/netcdf-c/pull/1784].
> - [Bug Fix] Don’t return error for extra calls to nc_redef() for
> netCDF/HDF5 files, unless classic model is in use. See [
> https://github.com/Unidata/netcdf-c/issues/1779].
> - [Enhancement] Added new parallel I/O benchmark program to mimic NOAA
> UFS data writes, built when –enable-benchmarks is in configure. See [
> https://github.com/Unidata/netcdf-c/pull/1777].
> - [Bug Fix] Now allow szip to be used on variables with unlimited
> dimension [https://github.com/Unidata/netcdf-c/issues/1774].
> - [Enhancement] Add support for cloud storage using a variant of the
> Zarr storage format. Warning: this feature is highly experimental and is
> subject to rapid evolution [
>
> https://www.unidata.ucar.edu/blogs/developer/en/entry/overview-of-zarr-support-in
> ].
> - [Bug Fix] Fix nccopy to properly set default chunking parameters
> when not otherwise specified. This can significantly improve performance in
> selected cases. Note that if seeing slow performance with nccopy, then, as
> a work-around, specifically set the chunking parameters. [
> https://github.com/Unidata/netcdf-c/issues/1763].
> - [Bug Fix] Fix some protocol bugs/differences between the netcdf-c
> library and the OPeNDAP Hyrax server. Also cleanup checksum handling [
> https://github.com/Unidata/netcdf-c/issues/1712].* [Bug Fix]
> IMPORTANT: Ncgen was not properly handling large
> data sections. The problem manifests as incorrect ordering of
> data in the created file. Aside from examining the file with
> ncdump, the error can be detected by running ncgen with the -lc
> flag (to produce a C file). Examine the file to see if any
> variable is written in pieces as opposed to a single call to
> nc_put_vara. If multiple calls to nc_put_vara are used to write
> a variable, then it is probable that the data order is
> incorrect. Such multiple writes can occur for large variables
> and especially when one of the dimensions is unlimited.
> - [Bug Fix] Add necessary _
> *declspec declarations to allow compilation of netcdf library without
> causing errors or (*declspec related)
> warnings [https://github.com/Unidata/netcdf-c/issues/1725].
> - [Enhancement] When a filter is applied twice with different
> parameters, then the second set is used for writing the dataset
> [https://github.com/Unidata/netcdf-c/issues/1713].
> - [Bug Fix] Now larger cache settings are used for sequential HDF5
> file creates/opens on parallel I/O capable builds; see Github #1716
> <https://github.com/Unidata/netcdf-c/issues/1716> for more information .
> - [Bug Fix] Add functions to libdispatch/dnotnc4.c to support
> dispatch table operations that should work for any dispatch
> table, even if they do not do anything; functions such as
> nc_inq_var_filter [https://github.com/Unidata/netcdf-c/issues/1693].
> - [Bug Fix] Fixed a scalar annotation error when scalar == 0; see Github
> #1707 <https://github.com/Unidata/netcdf-c/issues/1707> for more
> information.
> - [Bug Fix] Use proper CURLOPT values for VERIFYHOST and VERIFYPEER;
> the semantics for VERIFYHOST in particular changed. Documented in
> NUG/DAP2.md. See [https://github.com/Unidata/netcdf-c/issues/1684].
> - [Bug Fix][cmake] Correct an issue with parallel filter test logic in
> CMake-based builds.
> - [Bug Fix] Now allow nc_inq_var_deflate()/nc_inq_var_szip() to be
> called for all formats, not just HDF5. Non-HDF5 files return NC_NOERR and
> report no compression in use. This reverts behavior that was changed in the
> 4.7.4 release. See [https://github.com/Unidata/netcdf-c/issues/1691].
> - [Bug Fix] Compiling on a big-endian machine exposes some missing
> forward delcarations in dfilter.c.
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> netcdfgroup mailing list
> netcdfgroup@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
>
