The netCDF Operators NCO version 4.4.1 are ready.
http://nco.sf.net (Homepage)
http://dust.ess.uci.edu/nco (Homepage "mirror")
This release is mainly a bugfix release for power users. If you use
NCO's ncea, ncra, or ncrcat version 4.3.9-4.4.0 on
hundreds-to-thousands of input files per invocation, then you may
wish to upgrade. This release fixes a bug that could cause
ncea/ncra/ncrcat to fail on large numbers (> 256) of input files.
This is embarassing because NCO has always been designed to work with
arbitrary numbers of input files and we want power users to be
comfortable running it on hundreds of thousands of input files.
So we're releasing 4.4.1 to fix this bug for power users.
http://nco.sf.net#bug_ncra_no_fl_close
There are also a few new features: NCO now supports full path names
for dimension arguments to chunking (--cnk_dmn), and supports using
most-in-scope latitude/longitude for auxiliary coordinates (-X).
Work on NCO 4.4.2 is underway, focused on stability and speed.
There will be more netCDF4 mop-up and, possibly, improved HDF4
support, and cache manipulation for chunking.
Enjoy,
Charlie
"New stuff" in 4.4.1 summary (full details always in ChangeLog):
NEW FEATURES:
A. The -X option for auxiliary coordinates now works fully when
latitude and longitude coordinates are nested in group hierarchies.
The most in-scope lat/long are found and used for each variable.
http://nco.sf.net/nco.html#aux
B. Chunking options now support full group pathnames for dimensions.
These override chunking arguments with short names when a given
dimension matches both the long and short name arguments, e.g.,
ncks --cnk_dmn lat,2 --cnk_dmn /g1/lat,4 in.nc out.nc
causes all dimensions named "lat" to be chunked with size=2
except for the /g1/lat dimension which has chunksize=4.
http://nco.sf.net/nco.html#cnk
BUG FIXES:
A. ncra/ncea/ncrcat in NCO 4.3.9-4.4.0 did not close or remove input
files after use. This could cause NCO to exceed system limits on
maximum number of open files when hundreds--thousands of input
files were specified. Fixed.
http://nco.sf.net#bug_ncra_no_fl_close
B. NCO 4.4.0 could sometimes fail when asked to simultaneously
uncompress (-L 0) and to chunk (--cnk_dmn) a file. This seems to
have occurred only if the compressed variable was shuffled.
Problem reported by Ben. Fixed.
C. Fix build problem exposed by LLVM on Mac OS X 10.9 Mavericks
KNOWN ISSUES NOT YET FIXED:
This section of ANNOUNCE reports and reminds users of the
existence and severity of known, not yet fixed, problems.
These problems occur with NCO 4.4.1 built/tested with netCDF
4.3.1 on top of HDF5 hdf5-1.8.9 with these methods:
cd ~/nco;./configure --enable-netcdf4 # Configure mechanism -or-
cd ~/nco/bld;make dir;make allinone # Old Makefile mechanism
A. NOT YET FIXED (NCO problem)
Correctly read arrays of NC_STRING with embedded delimiters in
ncatted arguments
Demonstration:
ncatted -D 5 -O -a
new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc
~/foo.nc
ncks -m -C -v att_var ~/foo.nc
20130724: Verified problem still exists
TODO nco1102
Cause: NCO parsing of ncatted arguments is not sophisticated
enough to handle arrays of NC_STRINGS with embedded delimiters.
B. NOT YET FIXED (netCDF library problem)
Probe hidden attributes (chunking, compression) of HDF4 files
Demonstration:
ncdump -h -s ~/nco/data/hdf.hdf # (dies)
ncks -m ~/nco/data/hdf.hdf # (works by avoiding fatal calls)
20131230: Verified problem still exists
Cause: some libnetCDF library functions fail on HDF4 file inquiries.
Bug report filed: netCDF #HZY-708311 ncdump/netCDF4 segfaults probing
HDF4 file
Tracking tickets NCF-272, NCF-273
C. NOT YET FIXED (would require DAP protocol change?)
Unable to retrieve contents of variables including period '.' in name
Periods are legal characters in netCDF variable names.
Metadata are returned successfully, data are not.
DAP non-transparency: Works locally, fails through DAP server.
Demonstration:
ncks -O -C -D 3 -v var_nm.dot -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to
find variable
20130724: Verified problem still exists.
Stopped testing because inclusion of var_nm.dot broke all test scripts.
NB: Hard to fix since DAP interprets '.' as structure delimiter in
HTTP query string.
Bug report filed: https://www.unidata.ucar.edu/jira/browse/NCF-47
D. NOT YET FIXED (would require DAP protocol change)
Correctly read scalar characters over DAP.
DAP non-transparency: Works locally, fails through DAP server.
Problem, IMHO, is with DAP definition/protocol
Demonstration:
ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc
20120801: Verified problem still exists
Bug report not filed
Cause: DAP translates scalar characters into 64-element (this
dimension is user-configurable, but still...), NUL-terminated
strings so MD5 agreement fails
"Sticky" reminders:
A. Pre-built, up-to-date Debian Sid & Ubuntu packages:
http://nco.sf.net#debian
B. Pre-built Fedora and CentOS RPMs:
http://nco.sf.net#rpm
C. Pre-built Windows (native) and Cygwin binaries:
http://nco.sf.net#windows
D. Pre-built AIX binaries:
http://nco.sf.net#aix
E. NCO support for netCDF4 features is tracked at
http://nco.sf.net/nco.html#nco4
NCO supports netCDF4 atomic data types, compression, chunking, and
groups.
F. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g.,
HDF4: AMSR MERRA MODIS ...
HDF5: GLAS ICESat Mabel SBUV ...
HDF-EOS5: AURA HIRDLS OMI ...
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(
