The netCDF operators NCO version 4.2.6 are ready.
http://nco.sf.net (Homepage)
http://dust.ess.uci.edu/nco (Homepage "mirror")
This release continues to improve support of netCDF4 files in ncks and
ncecat, and add a sprinkle of new and convenient features to the rest
of NCO. ncks and ncecat now handle corner cases of netCDF4 groups,
dimensions, and coordinates. We perverted our own test file with
tortuous situations that all work, such as dimensions, coordinates,
and variables all being in separate groups.
ncks and ncecat now work well on many common HDF5 datasets, e.g.,
NASA AURA HIRDLS HDF-EOS5
NASA ICESat GLAS HDF5
NASA SBUV HDF5...
Work on NCO 4.2.7 is underway and focuses on netCDF4 support for
ncrename and ncbo operators, and on better support for Windows builds
(enabling DAP, UDUnits).
Enjoy,
Charlie and Pedro
"New stuff" in 4.2.6 summary (full details always in ChangeLog):
NEW FEATURES:
A. Regular expressions (rx's) now work on full paths, not just stubs.
Previously, rx's only worked on last components of full names.
Now rx's like ncks -v /cesm/temperature.+ can succeed.
http://nco.sf.net/nco.html#rx
B. ncks now counts and then ignores non-atomic type netCDF4 variables.
Previously, ncks would die if it encountered the new netCDF4
non-atomic types: vlen, opaque, enum, and compound.
Now ncks prints the variable name and then ignores it and processes
all the atomic-type (float, double, int, string...) variables as
usual.
C. ncks prints fulls paths of non-local dimensions and coordinates.
D. ncrename allows "mandatory attributes" (i.e., those not prefixed
with a period which would make them "optional attributes") to be
present in groups/global instead of in variables.
E. ncflint accepts a new switch, --fix_rec_crd, that prevents record
coordinates from being multiplied/interpolated. By default, ncflint
multplies/interpolates record coordinates (like time) but not other
coordinates (like, typically, latitude). This is because ncflint
is most often used to time-interpolate files (so you want the time
to change, but not the spatial coordinates). However, ncflint is
also used quite often to "scale" files by multiplying the variables
by a constant. Now one can easily multiply all variables (except
coordinates) by a constant, and if --fix_rec_crd is specified, the
record coordinate will not change.
BUG FIXES:
A. ncks correctly interprets scopes in corner cases including:
1. Dimension defined in ancestor of variable's group
2. Coordinate defined in relative (sibling, ancestor, descendent)
of variable's group
3. Single dimension has multiple associated coordinates
Handling these cases correctly is a major milestone for NCO.
B. ncks prints accurate netCDF4 metadata summaries (with -M) and
breaks dimensions into fixed vs. record, and variables into
atomic-types vs. non-atomic. Formerly, the number of netCDF4
dimensions was ill-defined.
C. ncks 4.2.4-4.2.5 failed when attempting to copy certain netCDF3
files (known as MM3 files because they contain multiple record
variables that require a special copying algorithm) into netCDF4
files when group-path-editing (GPE) was invoked. This has been
fixed so commands like this work again:
ncks -G g1 -4 in.nc out.nc
KNOWN BUGS NOT YET FIXED:
This section of the ANNOUNCE file reports and reminds users of the
existence and severity of known, not yet fixed, problems.
These problems occur with NCO 4.2.6 built/tested with netCDF
4.3.0-rc3 released 20130311 on top of HDF5 hdf5-1.8.9 with these
methods:
cd ~/nco;./configure --enable-netcdf4 # Configure mechanism -or-
cd ~/nco/bld;make dir;make all;make ncap2 # Old Makefile mechanism
A. NOT YET FIXED
Correctly read netCDF4 input over DAP, write netCDF4 output, then
read resulting file.
Replacing netCDF4 with netCDF3 in either location of preceding
sentence leads to success.
DAP non-transparency: Works locally, fails through DAP server.
Unclear whether resulting file is "legal" because of dimension ID
ordering assumptions
Demonstration:
ncks -4 -O -v three_dmn_rec_var
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods/in_4.nc ~/foo.nc
ncks ~/foo.nc # breaks with "NetCDF: Invalid dimension ID or name"
20120731: Unable to verify since in_4.nc no longer accessible
Bug report filed: netCDF #QUN-641037: dimension ID ordering assumptions
B. NOT YET FIXED
netCDF4 library fails when renaming dimension and variable using
that dimension, in either order. Works fine with netCDF3.
Problem with netCDF4 library implementation.
Demonstration:
ncks -O -4 -v lat_T42 ~/nco/data/in.nc ~/foo.nc
ncrename -O -D 2 -d lat_T42,lat -v lat_T42,lat ~/foo.nc ~/foo2.nc #
Breaks with "NetCDF: HDF error"
ncks -m ~/foo.nc
20130319: Verified problem still exists
Bug report filed: netCDF #YQN-334036: problem renaming dimension and
coordinate in netCDF4 file
C. NOT YET FIXED
Unable to retrieve contents of variables including period '.' in name
Metadata is returned successfully, data is not.
DAP non-transparency: Works locally, fails through DAP server.
Demonstration:
ncks -O -C -D 3 -v var_nm.dot -p
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods in.nc # Fails to
find variable
20120731: Verified problem still exists.
Stopped testing because inclusion of var_nm.dot broke all test scripts.
NB: Problem hard to fix since DAP interprets '.' as structure
delimiter in query string of HTTP requests.
Bug report filed: https://www.unidata.ucar.edu/jira/browse/NCF-47
D. NOT YET FIXED
Correctly read scalar characters over.
DAP non-transparency: Works locally, fails through DAP server.
Problem, IMHO, is with DAP definition/protocol
Demonstration:
ncks -O -D 1 -H -C -m --md5 -v md5_a -p
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods in.nc
20120801: Verified problem still exists
Bug report not filed
Cause: DAP translates scalar characters into 64-element,
NUL-terminated strings so MD5 agreement fails
E. NOT YET FIXED
netCDF4 library can create dimensions with duplicate IDs if
dimension with same name defined in a group and its ancestor group.
Problem with HDF5 or with netCDF4 library implementation?
Demonstration:
ncks -O -v two_dmn_rec_var ~/nco/data/in_grp.nc ~/foo.nc
ncks -m ~/foo.nc
20130319: Verified problem still exists
Bug report filed 20120312: netCDF #SHH-257980: Re: [netcdfgroup]
Dimensions IDs
F. NOT YET FIXED
ncdump is unable to dump the NCO test file in_grp.nc
Unclear whether problem with ncdump, HDF5, or with netCDF4 library
implementation
Demonstration:
ncdump ~/nco/data/in_grp.nc # breaks with "NetCDF: Invalid argument
Location: file dumplib.c; line 970"
20130319: Verified
Bug report not yet filed
"Sticky" reminders:
A. Pre-built, up-to-date Debian Sid & Ubuntu packages:
http://nco.sf.net#debian
B. Pre-built Fedora and CentOS RPMs:
http://nco.sf.net#rpm
C. Pre-built Windows (native) and Cygwin binaries:
http://nco.sf.net#windows
D. Pre-built AIX binaries:
http://nco.sf.net#aix
E. Did you try SWAMP (Script Workflow Analysis for MultiProcessing)?
SWAMP efficiently schedules/executes NCO scripts on remote servers:
http://swamp.googlecode.com
SWAMP can work command-line operator analysis scripts besides NCO.
If you must transfer lots of data from a server to your client
before you analyze it, then SWAMP will likely speed things up.
F. NCO support for netCDF4 features is tracked at
http://nco.sf.net/nco.html#nco4
NCO supports netCDF4 atomic data types, compression, chunking, and
groups.
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(
