The netCDF Operators NCO version 4.4.4 are ready.
http://nco.sf.net (Homepage)
http://dust.ess.uci.edu/nco (Homepage "mirror")
This release is mainly a bugfix and stability release.
A few new features improve some corner-cases:
It includes workarounds that allow NCO to successfully work with
some names that violate netCDF but not HDF rules.
Users can now retain "orphaned" dimensions after variable subsetting.
nces now works with hyperslabs that resolve to the same size.
Group broadcasting by ncbo is more intelligent when differencing
ensemble means (created by nces) from full datasets.
A potentially serious bug in ncwa --mask_condition has been fixed.
Users of ncwa --mask_condition are encouraged to upgrade.
Work on NCO 4.4.5 is underway, still focused on stability and speed.
This includes more netCDF4 mop-up, improved group broadcasting
algorithms, and, possibly, improved HDF4 support, and new chunking
features.
Enjoy,
Charlie
NEW FEATURES (full details always in ChangeLog):
A. NCO now contains automatic workarounds to handle element names that
contain characters that are legal in HDF though are illegal in
netCDF. For example, slashes and leading special characters are
are legal in HDF and illegal in netCDF element (i.e., group,
variable, dimension, and attribute) names. NCO converts these
forbidden characters to underscores, and retains the original names
of variables in automatically produced attributes named hdf_name.
Two real-world examples: NCO translates the NASA CERES dimension
"(FOV) Footprints" to "_FOV_ Footprints", and "Cloud & Aerosol,
Cloud Only, Clear Sky w/Aerosol, and Clear Sky" (yes, that's really
a dimension name) to "Cloud & Aerosol, Cloud Only, Clear Sky
w_Aerosol, and Clear Sky".
ncks CERES_MODIS.hdf CERES_MODIS.nc
http://nco.sf.net/nco.html#hdf4
B. ncks --rad to Retain all dimensions.
When invoked with --rad, ncks copies each dimension in the input
file to the output file, regardless of whether the dimension is
utilized by any variables. Normally ncks discards "orphan
dimensions", i.e., dimensions not referenced by any variables.
This switch allows users to keep non-referenced dimensions.
When invoked in printing mode, this causes orphaned dimensions to
be printed (they are not printed by default).
ncks -O -C -v lat,lon ~/nco/data/in.nc ~/foo.nc
ncks --rad -O -x -v lat ~/foo.nc ~/foo2.nc
ncks --rad --cdl ~/foo2.nc
http://nco.sf.net/nco.html#rad
C. nces now computes statistics on ensemble members whose specified
hyperslabs resolve to the same size. nces has always worked well
with identically shaped ensemble members, and on ensemble members
whose record-dimension hyperslabs resolve to the same size.
Now nces works well with ensemble members whose specified
hyperslabs along any dimension resolve to the same size. The
use-case that demanded this was a researcher who wished to compare
a certain lat/lon region (stored in some regional files) with the
same region stored in other global files (with the same grid).
The regional files (e.g., Greenland only) are much smaller than
global files so nces's new capability allows users to "get by" with
regional files rather than always storing global files.
nces -O --nsm_grp -d lat,0 -d lon,2 -d time,0 -p ~/nco/data grp1.nc
grp2.nc ~/foo.nc
http://nco.sf.net/nco.html#nces
BUG FIXES:
A. Fixed ncwa incorrect interpretation of some --mask_conditions.
Lexer/Parser could return incorrect mask values for negative numbers.
Thanks to Keith Lindsay for report, and to Henry Butowsky for fix.
Prior to this fix, the ncwa lexer would drop the negative sign, if
any, from the comparators appearing in the mask condition, e.g.,
ncwa --mask_condition "lat < -20" was parsed as "lat < 20" not
"lat < -20". Hence, users of ncwa --mask_condition (or -B) should
upgrade. NB: The -m -M -T form of ncwa masking is/was not buggy.
http://nco.sf.net#bug_ncwa_mask_condition
B. Restore former tolerance of -x for non-existent elements.
NCO normally requires that variables and groups specified with -v
and -g must match the input file contents. The exception is when
the -x flag tells NCO to exclude these elements. This exception
was broken since NCO 4.2.x, and is now working again. Now excluding
non-existent variables should one again work, e.g.,
ncks -O -x -v no_such_variable ~/nco/data/in.nc ~/foo.nc
Thanks to Keith Oleson for this bug report.
http://nco.sf.net/nco.html#x
C. Fix bug that caused CDL and XML printing to truncate trailing zeros
of floating point exponents. Before this bugfix, ncks --cdl would
print 1.100e-10 as 1.100e-1 instead of 1.1e-10. All CDL and XML
printing prior to 4.4.4 mis-prints floating point numbers with
exponents that end in zero (i.e., +/- 10, 20, 30 ... 200, 210...).
Users of CDL and XML mode are urged to upgrade. This bug changed
affected values by some multiple of 10 orders of magnitude.
Hence it is likely that users affected by this bug would notice it
when carefully performing any statistics, yet possible they would
not for fields with large dynamic ranges. Thanks to Stubaan for report.
D. Fix regression tests not to cause false negatives due to netCDF
4.3.2 record dimension chunk size changes
E. ncdismember better handles whitespace in group names
F. Fix numerous memory leaks and some corner-case segfaults
KNOWN PROBLEMS:
This section of ANNOUNCE reports and reminds users of the
existence and severity of known, not yet fixed, problems.
These problems occur with NCO 4.4.4 built/tested with netCDF
4.3.3-rc1 on top of HDF5 hdf5-1.8.12 with these methods:
cd ~/nco;./configure --enable-netcdf4 # Configure mechanism -or-
cd ~/nco/bld;make dir;make allinone # Old Makefile mechanism
A. NOT YET FIXED (NCO problem)
Correctly read arrays of NC_STRING with embedded delimiters in
ncatted arguments
Demonstration:
ncatted -D 5 -O -a
new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc
~/foo.nc
ncks -m -C -v att_var ~/foo.nc
20130724: Verified problem still exists
TODO nco1102
Cause: NCO parsing of ncatted arguments is not sophisticated
enough to handle arrays of NC_STRINGS with embedded delimiters.
B. NOT YET FIXED (netCDF library problem)
Probe hidden attributes (chunking, compression) of HDF4 files
Demonstration:
ncdump -h -s ~/nco/data/hdf.hdf # (dies)
ncks -m ~/nco/data/hdf.hdf # (works by avoiding fatal calls)
20131230: Verified problem still exists
Cause: some libnetCDF library functions fail on HDF4 file inquiries.
Bug report filed: netCDF #HZY-708311 ncdump/netCDF4 segfaults
probing HDF4 file
Tracking tickets NCF-272, NCF-273
C. NOT YET FIXED (would require DAP protocol change?)
Unable to retrieve contents of variables including period '.' in name
Periods are legal characters in netCDF variable names.
Metadata are returned successfully, data are not.
DAP non-transparency: Works locally, fails through DAP server.
Demonstration:
ncks -O -C -D 3 -v var_nm.dot -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to
find variable
20130724: Verified problem still exists.
Stopped testing because inclusion of var_nm.dot broke all test scripts.
NB: Hard to fix since DAP interprets '.' as structure delimiter in
HTTP query string.
Bug report filed: https://www.unidata.ucar.edu/jira/browse/NCF-47
D. NOT YET FIXED (would require DAP protocol change)
Correctly read scalar characters over DAP.
DAP non-transparency: Works locally, fails through DAP server.
Problem, IMHO, is with DAP definition/protocol
Demonstration:
ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc
20120801: Verified problem still exists
Bug report not filed
Cause: DAP translates scalar characters into 64-element (this
dimension is user-configurable, but still...), NUL-terminated
strings so MD5 agreement fails
"Sticky" reminders:
A. Pre-built Debian Sid & Ubuntu packages:
http://nco.sf.net#debian
B. Pre-built Fedora and CentOS RPMs:
http://nco.sf.net#rpm
C. Pre-built Mac binaries:
http://nco.sf.net#mac
D. Pre-built Windows (native) and Cygwin binaries:
http://nco.sf.net#windows
F. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g.,
HDF4: AMSR MERRA MODIS ...
HDF5: GLAS ICESat Mabel SBUV ...
HDF-EOS5: AURA HIRDLS OMI ...
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(
