The netCDF operators NCO version 4.2.4 are ready.
http://nco.sf.net (Homepage)
http://dust.ess.uci.edu/nco (Homepage "mirror")
This release continues, and largely concludes, the introduction of
powerful new sub-setting features aimed to give users powerful and
fine-grained control over subsetting hierarchical netCDF4 files.
Last release saw the introduction of Group Path Editing (GPE) which
controls the output location of extracted variables and groups.
This release introduces three new features that control extraction:
1. Group regular expressions, 2. Anchoring and Recursion, and
3. Intersection and Unions. These latest three features are only in
ncks and ncecat for the time-being.
Again, let us know how you like or dislike the way we have implemented
the netCDF4 group functionality in ncks and ncecat.
There are many choices for handling (selecting, printing, moving,
comparing) groups and we made what I would call tentative decisions.
Your feedback will help us finalize details like the printed output
format for group information and the syntax of switches.
This improved netCDF4 support will eventually come to all of NCO.
Work on NCO 4.2.5 is underway: better netCDF4 support for ncrename
and ncbo operators.
Enjoy,
Charlie
"New stuff" in 4.2.4 summary (full details always in ChangeLog):
NEW FEATURES:
A. ncks & ncecat support regular expressions in both variable and
group lists. Formerly they worked only in variable lists.
ncks -g 'g.' in_grp.nc # Groups g0-g9
ncks -g 'g.' -v 'v.' in_grp.nc # All variables 'v.' in groups 'g.'
http://nco.sf.net/nco.html#rx
B. ncks & ncecat support Anchoring and Recursion.
Previously arguments to -g were extracted recursively by default.
Meaning that descendents of specified groups were also extracted.
And prepended slashes had no special significance.
Now initial and trailing slashes control Anchoring and Recursion:
ncks -g g1 # Extract, recursively, all groups with a g1 component
ncks -g g1/ # Extract, non-recursively, all groups terminating in g1
ncks -g /g1 # Extract, recursively, root group g1
ncks -g /g1/ # Extract, non-recursively root group g1
http://nco.sf.net/nco.html#rcr
C. ncks & ncecat allow control over intersection- and union-subsetting.
Do users want intersections or unions of matches to the -v and -g
subsetting instructions? NCO is no longer wishy-washy: Intersection
mode is the default, and --unn selects union-mode:
ncks -g g1 -v v1 # Yes: /g1/v1, /g2/g1/v1. No: /v1, /g2/v1
ncks -g g1 -v v1 --unn # All vars in g1 (or descendents), or named v1
http://nco.sf.net/nco.html#unn
D. ncrename supports restricting actions to global/group attributes
Formerly ncrename arguments like -a old_nm,new_nm had no way of
restricting actions to global/group attributes and changing them
required implicit assumptions about whether variables could have
attributes of the same name. Now attribute renaming options can
be restricted to global/group attributes either by leaving the
var_nm portion empty (and retaining the @), or by using var_nm=
global to indicated that att_nm refers to a global attribute.
ncrename -a global@old_att_nm,new_att_nm in.nc
ncrename -a @old_att_nm,new_att_nm in.nc
ncrename -a global@.old_att_nm,new_att_nm in.nc
http://nco.sf.net/nco.html#xmp_ncrename
BUG FIXES:
A. Group attributes are now copied into all groups which are ancestors
of groups with extracted variables. Formerly, group attributes were
only copied into groups which themselves had extracted variables.
Now otherwise empty (i.e., no variables) intermediate groups will
have their attributes copied.
A. Fix bug introduced in 4.2.1 that causes ncea to break when handling
files that do not contain a record dimension. Thanks to Martin Otte
for reporting the bug and providing a working fix.
http://nco.sf.net#bug_ncea_rec_var
A. Fix bug introduced in 4.2.1 where ncra did not write results of
final normalization if trailing file(s) was/were superfluous.
In the most common case, all zeros appeared in the output file.
http://nco.sf.net#bug_ncra_flg_bfr_nrm
A. Fix 4.2.2 and 4.2.3 bug that kept NCO from building from source
when netCDF version 3 library was not available.
A. Handle variables specified in CF "bounds" and "coordinates"
attributes as coordinates in all ways. Specifically, dimension-
reducing operations (e.g., min/max/ttl) no longer operate on
CF conventions (since performing these operations on grids
usually makes no sense). Instead, the same dimension-reducing
operation applied to the coordinate (averaging) is now applied to
any CF "bounds" and "coordinates" variables.
KNOWN BUGS NOT YET FIXED:
This section of the ANNOUNCE file reports and reminds users of the
existence and severity of known, not yet fixed, problems.
These problems occur with NCO 4.2.4 built/tested with netCDF
4.2.1 on top of HDF5 hdf5-1.8.9 with these methods:
cd ~/nco;./configure --enable-netcdf4 # Configure mechanism -or-
cd ~/nco/bld;make dir;make all;make ncap2 # Old Makefile mechanism
A. NOT YET FIXED
Correctly read netCDF4 input over DAP, write netCDF4 output, then
read resulting file.
Replacing netCDF4 with netCDF3 in either location of preceding
sentence leads to success.
DAP non-transparency: Works locally, fails through DAP server.
Unclear whether resulting file is "legal" because of dimension ID
ordering assumptions
Demonstration:
ncks -4 -O -v three_dmn_rec_var
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods/in_4.nc ~/foo.nc
ncks ~/foo.nc # breaks with "NetCDF: Invalid dimension ID or name"
20120731: Unable to verify since in_4.nc no longer accessible
Bug report filed: netCDF #QUN-641037: dimension ID ordering assumptions
B. NOT YET FIXED
netCDF4 library fails when renaming dimension and variable using
that dimension, in either order. Works fine with netCDF3.
Problem with netCDF4 library implementation.
Demonstration:
ncks -O -4 -v lat_T42 ~/nco/data/in.nc ~/foo.nc
ncrename -O -D 2 -d lat_T42,lat -v lat_T42,lat ~/foo.nc ~/foo2.nc #
Breaks with "NetCDF: HDF error"
ncks -m ~/foo.nc
20121025: Verified problem still exists
Bug report filed: netCDF #YQN-334036: problem renaming dimension and
coordinate in netCDF4 file
C. NOT YET FIXED
Unable to retrieve contents of variables including period '.' in name
Metadata is returned successfully, data is not.
DAP non-transparency: Works locally, fails through DAP server.
Demonstration:
ncks -O -C -D 3 -v var_nm.dot -p
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods in.nc # Fails to
find variable
20120731: Verified problem still exists.
Stopped testing because inclusion of var_nm.dot broke all test scripts.
NB: Problem hard to fix since DAP interprets '.' as structure
delimiter in query string of HTTP requests.
Bug report filed: https://www.unidata.ucar.edu/jira/browse/NCF-47
D. NOT YET FIXED
Correctly read scalar characters over.
DAP non-transparency: Works locally, fails through DAP server.
Problem, IMHO, is with DAP definition/protocol
Demonstration:
ncks -O -D 1 -H -C -m --md5 -v md5_a -p
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods in.nc
20120801: Verified problem still exists
Bug report not filed
Cause: DAP translates scalar characters into 64-element,
NUL-terminated strings so MD5 agreement fails
"Sticky" reminders:
A. Pre-built, up-to-date Debian Sid & Ubuntu packages:
http://nco.sf.net#debian
B. Pre-built Fedora and CentOS RPMs:
http://nco.sf.net#rpm
C. Pre-built Windows (native) and Cygwin binaries:
http://nco.sf.net#windows
D. Pre-built AIX binaries:
http://nco.sf.net#aix
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(
