The netCDF Operators NCO version 4.3.8 are ready.
http://nco.sf.net (Homepage)
http://dust.ess.uci.edu/nco (Homepage "mirror")
ncatted now supports groups. That's the main new feature.
This completes NCO's migration from flat- to group-oriented files.
All core operators are now "group-aware" and support netCDF4 groups.
Most relevant options may support full or partial group paths.
Remaining options that do not will be migrated as time permits.
The exception to this is ncap2. Porting ncap2 to support groups
will be undertaken as resources permit. The main ncap2 developer
is a volunteer for whom we are trying to secure funding.
Work on NCO 4.3.9 is underway and includes improved netCDF4 support
for various options (all -X, ncwa -w and -m), other netCDF4 mop-up,
and improved support for HDF4 files. And the operator nces, which
we will debut/describe at AGU (Friday 12/13 11:35 AM Moscone West 2010).
Enjoy,
Charlie
"New stuff" in 4.3.8 summary (full details always in ChangeLog):
NEW FEATURES:
A. ncatted accepts full group paths to variables and groups.
Previously ncatted worked only on the root group of netCDF4 files.
Now you can edit all attributes in group files, in all
permutations, i.e., for variables, for groups and the root group.
ncatted -a units,/g1/lon,o,c,"degrees_west" in_grp.nc
ncatted -a new_grp_att,group,c,sng,"new group attribute" in_grp.nc
http://nco.sf.net/nco.html#xmp_ncatted
B. Add -7 switch to write netCDF4-classic files.
This is simply a synonym for --fl_fmt=netcdf4_classic.
The reason (and mnemonic) for -7 is that NETCDF4_CLASSIC files
include great features of both netCDF3 (compatibility) and
netCDF4 (compression, chunking) and, well, 3+4=7 :)
Anyway, -7 is much easier to remember/type than the alternative.
http://nco.sf.net/nco.html#fl_fmt
BUG FIXES:
A. Reduced namespace conflicts with other shared libraries
B. Added missing --op_typ/--operation switches to ncra
KNOWN ISSUES NOT YET FIXED:
This section of ANNOUNCE reports and reminds users of the
existence and severity of known, not yet fixed, problems.
These problems occur with NCO 4.3.7 built/tested with netCDF
4.3.1-rc4 snapshot 20131106 on top of HDF5 hdf5-1.8.9 with these
methods:
cd ~/nco;./configure --enable-netcdf4 # Configure mechanism -or-
cd ~/nco/bld;make dir;make all;make ncap2 # Old Makefile mechanism
A. NOT YET FIXED
netCDF4 library fails when renaming dimension and variable using
that dimension, in either order. Works fine with netCDF3.
Hence coordinate renaming does not work with netCDF4 files.
Problem with netCDF4 library implementation.
Demonstration:
ncks -O -4 -v lat_T42 ~/nco/data/in.nc ~/foo.nc
ncrename -O -D 2 -d lat_T42,lat -v lat_T42,lat ~/foo.nc ~/foo2.nc #
Breaks with "NetCDF: HDF error"
ncks -m ~/foo.nc
20130724: Verified problem still exists
Bug report filed: netCDF #YQN-334036: problem renaming dimension and
coordinate in netCDF4 file
Workaround: Use ncrename twice; first rename the variable, then
rename the dimension.
B. NOT YET FIXED (would require DAP protocol change?)
Unable to retrieve contents of variables including period '.' in name
Periods are legal characters in netCDF variable names.
Metadata are returned successfully, data are not.
DAP non-transparency: Works locally, fails through DAP server.
Demonstration:
ncks -O -C -D 3 -v var_nm.dot -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to
find variable
20130724: Verified problem still exists.
Stopped testing because inclusion of var_nm.dot broke all test scripts.
NB: Hard to fix since DAP interprets '.' as structure delimiter in
HTTP query string.
Bug report filed: https://www.unidata.ucar.edu/jira/browse/NCF-47
C. NOT YET FIXED (would require DAP protocol change)
Correctly read scalar characters over DAP.
DAP non-transparency: Works locally, fails through DAP server.
Problem, IMHO, is with DAP definition/protocol
Demonstration:
ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc
20120801: Verified problem still exists
Bug report not filed
Cause: DAP translates scalar characters into 64-element (this
dimension is user-configurable, but still...), NUL-terminated
strings so MD5 agreement fails
D. NOT YET FIXED (NCO problem)
Correctly read arrays of NC_STRING with embedded delimiters in
ncatted arguments
Demonstration:
ncatted -D 5 -O -a
new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc
~/foo.nc
20130724: Verified problem still exists
TODO nco1102
Cause: NCO parsing of ncatted arguments is not yet sophisticated
enough to handle arrays of NC_STRINGS with embedded delimiters.
E. NOT YET FIXED
Report correct chunking and compression information for HDF4 files
Demonstration:
ncdump -h -s MOP01-20121231-L1V3.34.10.hdf
ncks -m MOP01-20121231-L1V3.34.10.hdf
20131007: Verified problem still exists
Cause: some libnetCDF library functions fail on HDF4 file inquiries.
Bug report filed: netCDF #HZY-708311 ncdump/netCDF4 segfaults probing
HDF4 file
Tracking tickets NCF-272, NCF-273
"Sticky" reminders:
A. Pre-built, up-to-date Debian Sid & Ubuntu packages:
http://nco.sf.net#debian
B. Pre-built Fedora and CentOS RPMs:
http://nco.sf.net#rpm
C. Pre-built Windows (native) and Cygwin binaries:
http://nco.sf.net#windows
D. Pre-built AIX binaries:
http://nco.sf.net#aix
E. Did you try SWAMP (Script Workflow Analysis for MultiProcessing)?
SWAMP efficiently schedules/executes NCO scripts on remote servers:
http://swamp.googlecode.com
SWAMP can work command-line operator analysis scripts besides NCO.
If you must transfer lots of data from a server to your client
before you analyze it, then SWAMP will likely speed things up.
F. NCO support for netCDF4 features is tracked at
http://nco.sf.net/nco.html#nco4
NCO supports netCDF4 atomic data types, compression, chunking, and
groups.
G. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g.,
NASA AURA HIRDLS HDF-EOS5
NASA ICESat GLAS HDF5
NASA MERRA HDF4
NASA MODIS HDF4
NASA SBUV HDF5...
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(
