[netcdfgroup] netCDF operators NCO version 4.2.4 are ready

The netCDF operators NCO version 4.2.4 are ready.

http://nco.sf.net (Homepage)
http://dust.ess.uci.edu/nco (Homepage "mirror")

This release continues, and largely concludes, the introduction of
powerful new sub-setting features aimed to give users powerful and
fine-grained control over subsetting hierarchical netCDF4 files.
Last release saw the introduction of Group Path Editing (GPE) which
controls the output location of extracted variables and groups.
This release introduces three new features that control extraction:
1. Group regular expressions, 2. Anchoring and Recursion, and
3. Intersection and Unions. These latest three features are only in
ncks and ncecat for the time-being.

Again, let us know how you like or dislike the way we have implemented
the netCDF4 group functionality in ncks and ncecat.
There are many choices for handling (selecting, printing, moving,
comparing) groups and we made what I would call tentative decisions.
Your feedback will help us finalize details like the printed output
format for group information and the syntax of switches.
This improved netCDF4 support will eventually come to all of NCO.

Work on NCO 4.2.5 is underway: better netCDF4 support for ncrename
and ncbo operators.

Enjoy,
Charlie

"New stuff" in 4.2.4 summary (full details always in ChangeLog):

NEW FEATURES:

A. ncks & ncecat support regular expressions in both variable and
   group lists. Formerly they worked only in variable lists.
   ncks -g 'g.' in_grp.nc # Groups g0-g9
   ncks -g 'g.' -v 'v.' in_grp.nc # All variables 'v.' in groups 'g.'
   http://nco.sf.net/nco.html#rx

B. ncks & ncecat support Anchoring and Recursion.
   Previously arguments to -g were extracted recursively by default.
   Meaning that descendents of specified groups were also extracted.
   And prepended slashes had no special significance.
   Now initial and trailing slashes control Anchoring and Recursion:
   ncks -g  g1  # Extract, recursively, all groups with a g1 component
   ncks -g  g1/ # Extract, non-recursively, all groups terminating in g1
   ncks -g /g1  # Extract, recursively, root group g1
   ncks -g /g1/ # Extract, non-recursively root group g1
   http://nco.sf.net/nco.html#rcr

C. ncks & ncecat allow control over intersection- and union-subsetting.
   Do users want intersections or unions of matches to the -v and -g
   subsetting instructions? NCO is no longer wishy-washy: Intersection
   mode is the default, and --unn selects union-mode:
   ncks -g g1 -v v1 # Yes: /g1/v1, /g2/g1/v1. No: /v1, /g2/v1
   ncks -g g1 -v v1 --unn # All vars in g1 (or descendents), or named v1
   http://nco.sf.net/nco.html#unn

D. ncrename supports restricting actions to global/group attributes
   Formerly ncrename arguments like -a old_nm,new_nm had no way of
   restricting actions to global/group attributes and changing them
   required implicit assumptions about whether variables could have
   attributes of the same name. Now attribute renaming options can
   be restricted to global/group attributes either by leaving the
   var_nm portion empty (and retaining the @), or by using var_nm=
   global to indicated that att_nm refers to a global attribute.
   ncrename -a  global@old_att_nm,new_att_nm in.nc
   ncrename -a        @old_att_nm,new_att_nm in.nc
   ncrename -a global@.old_att_nm,new_att_nm in.nc
   http://nco.sf.net/nco.html#xmp_ncrename

BUG FIXES:

A. Group attributes are now copied into all groups which are ancestors
   of groups with extracted variables. Formerly, group attributes were
   only copied into groups which themselves had extracted variables.
   Now otherwise empty (i.e., no variables) intermediate groups will
   have their attributes copied.

A. Fix bug introduced in 4.2.1 that causes ncea to break when handling
   files that do not contain a record dimension. Thanks to Martin Otte
   for reporting the bug and providing a working fix.
   http://nco.sf.net#bug_ncea_rec_var

A. Fix bug introduced in 4.2.1 where ncra did not write results of
   final normalization if trailing file(s) was/were superfluous.
   In the most common case, all zeros appeared in the output file.
   http://nco.sf.net#bug_ncra_flg_bfr_nrm

A. Fix 4.2.2 and 4.2.3 bug that kept NCO from building from source
   when netCDF version 3 library was not available.

A. Handle variables specified in CF "bounds" and "coordinates"
   attributes as coordinates in all ways. Specifically, dimension-
   reducing operations (e.g., min/max/ttl) no longer operate on
   CF conventions (since performing these operations on grids
   usually makes no sense). Instead, the same dimension-reducing
   operation applied to the coordinate (averaging) is now applied to
   any CF "bounds" and "coordinates" variables.

KNOWN BUGS NOT YET FIXED:

   This section of the ANNOUNCE file reports and reminds users of the
   existence and severity of known, not yet fixed, problems.
   These problems occur with NCO 4.2.4 built/tested with netCDF
   4.2.1 on top of HDF5 hdf5-1.8.9 with these methods:

   cd ~/nco;./configure --enable-netcdf4  # Configure mechanism -or-
   cd ~/nco/bld;make dir;make all;make ncap2 # Old Makefile mechanism

A. NOT YET FIXED
   Correctly read netCDF4 input over DAP, write netCDF4 output, then
read resulting file.
   Replacing netCDF4 with netCDF3 in either location of preceding
sentence leads to success.
   DAP non-transparency: Works locally, fails through DAP server.
   Unclear whether resulting file is "legal" because of dimension ID
ordering assumptions

   Demonstration:
   ncks -4 -O -v three_dmn_rec_var
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods/in_4.nc ~/foo.nc
   ncks ~/foo.nc # breaks with "NetCDF: Invalid dimension ID or name"

   20120731: Unable to verify since in_4.nc no longer accessible
   Bug report filed: netCDF #QUN-641037: dimension ID ordering assumptions

B. NOT YET FIXED
   netCDF4 library fails when renaming dimension and variable using
   that dimension, in either order. Works fine with netCDF3.
   Problem with netCDF4 library implementation.

   Demonstration:
   ncks -O -4 -v lat_T42 ~/nco/data/in.nc ~/foo.nc
   ncrename -O -D 2 -d lat_T42,lat -v lat_T42,lat ~/foo.nc ~/foo2.nc #
Breaks with "NetCDF: HDF error"
   ncks -m ~/foo.nc

   20121025: Verified problem still exists
   Bug report filed: netCDF #YQN-334036: problem renaming dimension and
coordinate in netCDF4 file

C. NOT YET FIXED
   Unable to retrieve contents of variables including period '.' in name
   Metadata is returned successfully, data is not.
   DAP non-transparency: Works locally, fails through DAP server.

   Demonstration:
   ncks -O -C -D 3 -v var_nm.dot -p
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods in.nc # Fails to
find variable

   20120731: Verified problem still exists.
   Stopped testing because inclusion of var_nm.dot broke all test scripts.
   NB: Problem hard to fix since DAP interprets '.' as structure
delimiter in query string of HTTP requests.

   Bug report filed: https://www.unidata.ucar.edu/jira/browse/NCF-47

D. NOT YET FIXED
   Correctly read scalar characters over.
   DAP non-transparency: Works locally, fails through DAP server.
   Problem, IMHO, is with DAP definition/protocol

   Demonstration:
   ncks -O -D 1 -H -C -m --md5 -v md5_a -p
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods in.nc

   20120801: Verified problem still exists
   Bug report not filed
   Cause: DAP translates scalar characters into 64-element,
NUL-terminated strings so MD5 agreement fails

"Sticky" reminders:

A. Pre-built, up-to-date Debian Sid & Ubuntu packages:
   http://nco.sf.net#debian

B. Pre-built Fedora and CentOS RPMs:
   http://nco.sf.net#rpm

C. Pre-built Windows (native) and Cygwin binaries:
   http://nco.sf.net#windows

D. Pre-built AIX binaries:
   http://nco.sf.net#aix

-- 
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(



  • 2013 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: