The netCDF Operators NCO version 4.3.3 are ready.
http://nco.sf.net (Homepage)
http://dust.ess.uci.edu/nco (Homepage "mirror")
The current release brings full group support to ncwa and ncrename.
Now seven operators fully support group hierarchies: ncbo, ncecat,
ncflint, ncks, ncpdq, ncrename, and ncwa.
ncks has a new, highly-configurable printing mode which displays data
from files with groups with friendlier indentation and annotation,
including in CDL and XML modes. The new formats involve many choices
for indentation and spacing. We would be happy to learn what format
changes would make analyzing your data easier. The old "traditional"
ncks printing format remains the default, though this may change in
the future. Please let us know if you find any substantive
(non-whitespace) differences outputting CDL from ncdump and ncks.
Work on NCO 4.3.4 is underway and includes improved netCDF4 support
for more NCO operators (ncatted, ncra, ncea, ncrcat) and improved
support for HDF4 files.
Enjoy,
Charlie
"New stuff" in 4.3.3 summary (full details always in ChangeLog):
NEW FEATURES:
A. ncwa now supports groups, Group Path Editing (GPE) and list unions,
the same options supported by ncbo, ncecat, ncflint, ncks, and ncpdq.
Features are controlled by -g, -G and --union, respectively.
ncwa -a lon -g g1 -v v1 -G dude -O -p ~/nco/data in_grp.nc ~/foo.nc
http://nco.sf.net/nco.html#ncwa
http://nco.sf.net/nco.html#gpe
http://nco.sf.net/nco.html#union
B. ncrename will rename groups when compiled (with
-DNC_HAVE_RENAME_GRP) against the latest version (~20130720) of
libnetcdf.a (which will eventually be released as version 4.3.1).
This is much faster than using GPE and ncks to rename groups.
Note that the location of the new group is the same as the old.
To move groups around, one still needs to use GPE.
ncrename -g g1,obama in_grp.nc out.nc
http://nco.sf.net/nco.html#ncrename
http://nco.sf.net/nco.html#gpe
C. ncks will write the MD5 digest of each variable as an NC_CHAR
attribute named MD5. Activate this with the --md5_wrt or
--md5_write_attribute long options.
ncks -O -C --md5_wrt -v md5_.? ~/nco/data/in.nc ~/out.nc
http://nco.sf.net/nco.html#md5
D. ncks -5 supports a new, group-friendly printing format.
This format uses indentation to indicate group level.
Groups are arranged alphabetically, and variable metadata and data
are printed alphabetically within each group. Otherwise the
information printed is in "traditional" ncks format (but see --cdl
switch below for another possibility). Feedback welcome!
ncks -5 ~/nco/data/in_grp.nc
http://nco.sf.net/nco.html#ncks
E. ncks --cdl prints results in CDL format. CDL is the human-readable
lingua franca of netCDF imported by ncgen and exported by ncdump.
"Traditional" ncks printing was designed for flat netCDF3 files,
and for examining hyperslabs of data values. CDL printing is
friendlier for examining group hierarchies, and the CDL output
is directly usable by ncgen. Moreover, ncks outputs the minimal
CDL that matches the user's request, and CDL mode plays well with
other ncks options like hyperslabs:
ncks --cdl ~/nco/data/in_grp.nc
ncks --cdl ~/nco/data/in.nc
ncks --cdl -v v1 ~/nco/data/in_grp.nc
ncks --cdl -v three_dmn_rec_var -d lon,2 ~/nco/data/in_grp.nc
http://nco.sf.net/nco.html#cdl
F. ncks --xml prints metadata in NcML XML format.
NcML is also exported by ncdump -x.
ncks outputs the minimal NcML that matches the user's request.
ncks --xml ~/nco/data/in_grp.nc
ncks --xml ~/nco/data/in.nc
ncks --xml -v v1 ~/nco/data/in_grp.nc
http://nco.sf.net/nco.html#xml
BUG FIXES:
A. Fixed potential ncpdq bug packing variables in subgroups.
B. ncatted supports attributes of type NC_STRING.
Earlier versions of ncatted fail when handling NC_STRINGs.
ncatted -a string_att,att_var,c,sng,"string, cheese" in.nc out.nc
http://nco.sf.net/nco.html#ncatted
KNOWN BUGS NOT YET FIXED:
This section of ANNOUNCE reports and reminds users of the
existence and severity of known, not yet fixed, problems.
These problems occur with NCO 4.3.3 built/tested with netCDF
4.3.0 on top of HDF5 hdf5-1.8.9 with these methods:
cd ~/nco;./configure --enable-netcdf4 # Configure mechanism -or-
cd ~/nco/bld;make dir;make all;make ncap2 # Old Makefile mechanism
A. NOT YET FIXED
Correctly read netCDF4 input over DAP, write netCDF4 output, then
read resulting file.
Replacing netCDF4 with netCDF3 in either location of preceding
sentence leads to success.
DAP non-transparency: Works locally, fails through DAP server.
Unclear whether resulting file is "legal" because of dimension ID
ordering assumptions.
Demonstration:
ncks -4 -O -v three_dmn_rec_var
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods/in_4.nc ~/foo.nc
ncks ~/foo.nc # breaks with "NetCDF: Invalid dimension ID or name"
20130724: Unable to verify since in_4.nc no longer accessible on
Unidata DAP server
Bug report filed: netCDF #QUN-641037: dimension ID ordering assumptions
B. NOT YET FIXED
netCDF4 library fails when renaming dimension and variable using
that dimension, in either order. Works fine with netCDF3.
Problem with netCDF4 library implementation.
Demonstration:
ncks -O -4 -v lat_T42 ~/nco/data/in.nc ~/foo.nc
ncrename -O -D 2 -d lat_T42,lat -v lat_T42,lat ~/foo.nc ~/foo2.nc #
Breaks with "NetCDF: HDF error"
ncks -m ~/foo.nc
20130724: Verified problem still exists
Bug report filed: netCDF #YQN-334036: problem renaming dimension and
coordinate in netCDF4 file
C. NOT YET FIXED (requires change to DAP protocol)
Unable to retrieve contents of variables including period '.' in name
Periods are legal characters in netCDF variable names.
Metadata are returned successfully, data are not.
DAP non-transparency: Works locally, fails through DAP server.
Demonstration:
ncks -O -C -D 3 -v var_nm.dot -p
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods in.nc # Fails to
find variable
20130724: Verified problem still exists.
Stopped testing because inclusion of var_nm.dot broke all test scripts.
NB: Hard to fix since DAP interprets '.' as structure delimiter in
HTTP query string.
Bug report filed: https://www.unidata.ucar.edu/jira/browse/NCF-47
D. NOT YET FIXED (requires change to DAP protocol)
Correctly read scalar characters over DAP.
DAP non-transparency: Works locally, fails through DAP server.
Problem, IMHO, is with DAP definition/protocol
Demonstration:
ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods in.nc
20120801: Verified problem still exists
Bug report not filed
Cause: DAP translates scalar characters into 64-element (this
dimension is user-configurable, but still...), NUL-terminated
strings so MD5 agreement fails
E. NOT YET FIXED (NCO problem)
Correctly read arrays of NC_STRING in ncatted arguments
Demonstration:
ncatted -D 5 -O -a
new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc
~/foo.nc
20130724: Verified problem still exists
TODO nco1102
Cause: NCO parsing of ncatted arguments is not yet sophisticated
enough to handle arrays of NC_STRINGS with embedded delimiters.
"Sticky" reminders:
A. Pre-built, up-to-date Debian Sid & Ubuntu packages:
http://nco.sf.net#debian
B. Pre-built Fedora and CentOS RPMs:
http://nco.sf.net#rpm
C. Pre-built Windows (native) and Cygwin binaries:
http://nco.sf.net#windows
D. Pre-built AIX binaries:
http://nco.sf.net#aix
E. Did you try SWAMP (Script Workflow Analysis for MultiProcessing)?
SWAMP efficiently schedules/executes NCO scripts on remote servers:
http://swamp.googlecode.com
SWAMP can work command-line operator analysis scripts besides NCO.
If you must transfer lots of data from a server to your client
before you analyze it, then SWAMP will likely speed things up.
F. NCO support for netCDF4 features is tracked at
http://nco.sf.net/nco.html#nco4
NCO supports netCDF4 atomic data types, compression, chunking, and
groups.
G. Reminder that ncks, ncecat, ncbo, ncflint, and ncpdq work on many common
HDF5 datasets, e.g.,
NASA AURA HIRDLS HDF-EOS5
NASA ICESat GLAS HDF5
NASA SBUV HDF5...
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(
