[netcdfgroup] netCDF Operators NCO version 4.4.4 are ready

The netCDF Operators NCO version 4.4.4 are ready.

http://nco.sf.net (Homepage)
http://dust.ess.uci.edu/nco (Homepage "mirror")

This release is mainly a bugfix and stability release.
A few new features improve some corner-cases:
It includes workarounds that allow NCO to successfully work with
some names that violate netCDF but not HDF rules.
Users can now retain "orphaned" dimensions after variable subsetting.
nces now works with hyperslabs that resolve to the same size.
Group broadcasting by ncbo is more intelligent when differencing
ensemble means (created by nces) from full datasets.

A potentially serious bug in ncwa --mask_condition has been fixed.
Users of ncwa --mask_condition are encouraged to upgrade.

Work on NCO 4.4.5 is underway, still focused on stability and speed.
This includes more netCDF4 mop-up, improved group broadcasting
algorithms, and, possibly, improved HDF4 support, and new chunking
features.

Enjoy,
Charlie

NEW FEATURES (full details always in ChangeLog):

A. NCO now contains automatic workarounds to handle element names that
   contain characters that are legal in HDF though are illegal in
   netCDF.  For example, slashes and leading special characters are
   are legal in HDF and illegal in netCDF element (i.e., group,
   variable, dimension, and attribute) names. NCO converts these
   forbidden characters to underscores, and retains the original names
   of variables in automatically produced attributes named hdf_name.
   Two real-world examples: NCO translates the NASA CERES dimension
   "(FOV) Footprints" to "_FOV_ Footprints", and "Cloud & Aerosol,
   Cloud Only, Clear Sky w/Aerosol, and Clear Sky" (yes, that's really
   a dimension name) to "Cloud & Aerosol, Cloud Only, Clear Sky
   w_Aerosol, and Clear Sky".
   ncks CERES_MODIS.hdf CERES_MODIS.nc
   http://nco.sf.net/nco.html#hdf4

B. ncks --rad to Retain all dimensions.
   When invoked with --rad, ncks copies each dimension in the input
   file to the output file, regardless of whether the dimension is
   utilized by any variables. Normally ncks discards "orphan
   dimensions", i.e., dimensions not referenced by any variables.
   This switch allows users to keep non-referenced dimensions.
   When invoked in printing mode, this causes orphaned dimensions to
   be printed (they are not printed by default).
   ncks -O -C -v lat,lon ~/nco/data/in.nc ~/foo.nc
   ncks --rad -O -x -v lat ~/foo.nc ~/foo2.nc
   ncks --rad --cdl ~/foo2.nc
   http://nco.sf.net/nco.html#rad

C. nces now computes statistics on ensemble members whose specified
   hyperslabs resolve to the same size. nces has always worked well
   with identically shaped ensemble members, and on ensemble members
   whose record-dimension hyperslabs resolve to the same size.
   Now nces works well with ensemble members whose specified
   hyperslabs along any dimension resolve to the same size. The
   use-case that demanded this was a researcher who wished to compare
   a certain lat/lon region (stored in some regional files) with the
   same region stored in other global files (with the same grid).
   The regional files (e.g., Greenland only) are much smaller than
   global files so nces's new capability allows users to "get by" with
   regional files rather than always storing global files.
nces -O --nsm_grp -d lat,0 -d lon,2 -d time,0 -p ~/nco/data grp1.nc grp2.nc ~/foo.nc
   http://nco.sf.net/nco.html#nces

BUG FIXES:

A. Fixed ncwa incorrect interpretation of some --mask_conditions.
   Lexer/Parser could return incorrect mask values for negative numbers.
   Thanks to Keith Lindsay for report, and to Henry Butowsky for fix.
   Prior to this fix, the ncwa lexer would drop the negative sign, if
   any, from the comparators appearing in the mask condition, e.g.,
   ncwa --mask_condition "lat < -20" was parsed as "lat < 20" not
   "lat < -20". Hence, users of ncwa --mask_condition (or -B) should
   upgrade. NB: The -m -M -T form of ncwa masking is/was not buggy.
   http://nco.sf.net#bug_ncwa_mask_condition

B. Restore former tolerance of -x for non-existent elements.
   NCO normally requires that variables and groups specified with -v
   and -g must match the input file contents. The exception is when
   the -x flag tells NCO to exclude these elements. This exception
   was broken since NCO 4.2.x, and is now working again. Now excluding
   non-existent variables should one again work, e.g.,
   ncks -O -x -v no_such_variable ~/nco/data/in.nc ~/foo.nc
   Thanks to Keith Oleson for this bug report.
   http://nco.sf.net/nco.html#x

C. Fix bug that caused CDL and XML printing to truncate trailing zeros
   of floating point exponents. Before this bugfix, ncks --cdl would
   print 1.100e-10 as 1.100e-1 instead of 1.1e-10. All CDL and XML
   printing prior to 4.4.4 mis-prints floating point numbers with
   exponents that end in zero (i.e., +/- 10, 20, 30 ... 200, 210...).
   Users of CDL and XML mode are urged to upgrade. This bug changed
   affected values by some multiple of 10 orders of magnitude.
   Hence it is likely that users affected by this bug would notice it
   when carefully performing any statistics, yet possible they would
   not for fields with large dynamic ranges. Thanks to Stubaan for report.

D. Fix regression tests not to cause false negatives due to netCDF
   4.3.2 record dimension chunk size changes

E. ncdismember better handles whitespace in group names

F. Fix numerous memory leaks and some corner-case segfaults

KNOWN PROBLEMS:

   This section of ANNOUNCE reports and reminds users of the
   existence and severity of known, not yet fixed, problems.
   These problems occur with NCO 4.4.4 built/tested with netCDF
   4.3.3-rc1 on top of HDF5 hdf5-1.8.12 with these methods:

   cd ~/nco;./configure --enable-netcdf4  # Configure mechanism -or-
   cd ~/nco/bld;make dir;make allinone # Old Makefile mechanism

A. NOT YET FIXED (NCO problem)
Correctly read arrays of NC_STRING with embedded delimiters in ncatted arguments

   Demonstration:
ncatted -D 5 -O -a new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc ~/foo.nc
   ncks -m -C -v att_var ~/foo.nc

   20130724: Verified problem still exists
   TODO nco1102
   Cause: NCO parsing of ncatted arguments is not sophisticated
   enough to handle arrays of NC_STRINGS with embedded delimiters.

B. NOT YET FIXED (netCDF library problem)
   Probe hidden attributes (chunking, compression) of HDF4 files

   Demonstration:
   ncdump -h -s ~/nco/data/hdf.hdf # (dies)
   ncks -m ~/nco/data/hdf.hdf # (works by avoiding fatal calls)

   20131230: Verified problem still exists

   Cause: some libnetCDF library functions fail on HDF4 file inquiries.
Bug report filed: netCDF #HZY-708311 ncdump/netCDF4 segfaults probing HDF4 file
   Tracking tickets NCF-272, NCF-273

C. NOT YET FIXED (would require DAP protocol change?)
   Unable to retrieve contents of variables including period '.' in name
   Periods are legal characters in netCDF variable names.
   Metadata are returned successfully, data are not.
   DAP non-transparency: Works locally, fails through DAP server.

   Demonstration:
ncks -O -C -D 3 -v var_nm.dot -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to find variable

   20130724: Verified problem still exists.
   Stopped testing because inclusion of var_nm.dot broke all test scripts.
NB: Hard to fix since DAP interprets '.' as structure delimiter in HTTP query string.

   Bug report filed: https://www.unidata.ucar.edu/jira/browse/NCF-47

D. NOT YET FIXED (would require DAP protocol change)
   Correctly read scalar characters over DAP.
   DAP non-transparency: Works locally, fails through DAP server.
   Problem, IMHO, is with DAP definition/protocol

   Demonstration:
ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc

   20120801: Verified problem still exists
   Bug report not filed
   Cause: DAP translates scalar characters into 64-element (this
   dimension is user-configurable, but still...), NUL-terminated
   strings so MD5 agreement fails

"Sticky" reminders:

A. Pre-built Debian Sid & Ubuntu packages:
   http://nco.sf.net#debian

B. Pre-built Fedora and CentOS RPMs:
   http://nco.sf.net#rpm

C. Pre-built Mac binaries:
   http://nco.sf.net#mac

D. Pre-built Windows (native) and Cygwin binaries:
   http://nco.sf.net#windows

F. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g.,
   HDF4: AMSR MERRA MODIS ...
   HDF5: GLAS ICESat Mabel SBUV ...
   HDF-EOS5: AURA HIRDLS OMI ...

--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(



  • 2014 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: