The netCDF operators NCO version 4.2.3 are ready.
http://nco.sf.net (Homepage)
http://dust.ess.uci.edu/nco (Homepage "mirror")
This release occurs sooner than expected because it contains both
an important bufgix and a new feature:
We fixed a bug introduced in the last version, NCO 4.2.2, that could
cause ncecat, under limited circumstances, to skip a single file.
And we added a feature called Group Path Editing (GPE) that permits
users to move data en masse amongst arbitrary netCDF4 groups.
Again, let us know how you like or dislike the way we have implemented
the netCDF4 group functionality in ncks and ncecat.
Your feedback will help us finalize details like the printed output
format for group information and the syntax of switches.
This improved netCDF4 support will eventually come to all of NCO.
Work on NCO 4.2.4 is underway: better netCDF4 support for ncrename
and ncbo operators.
Enjoy,
Charlie
"New stuff" in 4.2.3 summary (full details always in ChangeLog):
NEW FEATURES:
A. ncks and ncecat support Group Path Editing (GPE):
The -G switch has been extended. Formerly it specified the root
group in the output file where to store data. In other words, full
output group paths were determined by appending the input group
path to the -G argument, if any. This now is a special case of GPE,
which handles appending, deleting, backspacing, and flattening
group paths. GPE examples:
ncks -G scenario1 in.nc out.nc # Put input file in group scenario1
ncks -G : in.nc out.nc # Flatten (remove groups from) input file
ncks -G :2 in.nc out.nc # Remove first two group levels
ncks -G g1:-2 in.nc out.nc # Replace last two groups levels by "g1"
ncks -G g2:1 -g g1 in.nc out.nc # Move data from g1 to g2
http://nco.sf.net/nco.html#gpe
BUG FIXES:
A. ncecat: Fix 4.2.2 bug that can skip first input file in the default
mode (RECORD_AGGREGATE) when -n NINTAP switch used to generate
filenames.
B. ncks correctly copies group attributes
KNOWN BUGS NOT YET FIXED:
This section of the ANNOUNCE file reports and reminds users of the
existence and severity of known, not yet fixed, problems.
These problems occur with NCO 4.2.3 built/tested with netCDF
4.2.1 on top of HDF5 hdf5-1.8.9 with these methods:
cd ~/nco;./configure --enable-netcdf4 # Configure mechanism -or-
cd ~/nco/bld;make dir;make all;make ncap2 # Old Makefile mechanism
A. NOT YET FIXED
Correctly read netCDF4 input over DAP, write netCDF4 output, then read
resulting file.
Replacing netCDF4 with netCDF3 in either location of preceding sentence
leads to success.
DAP non-transparency: Works locally, fails through DAP server.
Unclear whether resulting file is "legal" because of dimension ID
ordering assumptions
Demonstration:
ncks -4 -O -v three_dmn_rec_var
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods/in_4.nc ~/foo.nc
ncks ~/foo.nc # breaks with "NetCDF: Invalid dimension ID or name"
20120731: Unable to verify since in_4.nc no longer accessible
Bug report filed: netCDF #QUN-641037: dimension ID ordering assumptions
B. NOT YET FIXED
netCDF4 library fails when renaming dimension and variable using
that dimension, in either order. Works fine with netCDF3.
Problem with netCDF4 library implementation.
Demonstration:
ncks -O -4 -v lat_T42 ~/nco/data/in.nc ~/foo.nc
ncrename -O -D 2 -d lat_T42,lat -v lat_T42,lat ~/foo.nc ~/foo2.nc #
Breaks with "NetCDF: HDF error"
ncks -m ~/foo.nc
20121025: Verified problem still exists
Bug report filed: netCDF #YQN-334036: problem renaming dimension and
coordinate in netCDF4 file
C. NOT YET FIXED
Unable to retrieve contents of variables including period '.' in name
Metadata is returned successfully, data is not.
DAP non-transparency: Works locally, fails through DAP server.
Demonstration:
ncks -O -C -D 3 -v var_nm.dot -p
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods in.nc # Fails to
find variable
20120731: Verified problem still exists.
Stopped testing because inclusion of var_nm.dot broke all test scripts.
NB: Problem hard to fix since DAP interprets '.' as structure delimiter
in query string of HTTP requests.
Bug report filed: https://www.unidata.ucar.edu/jira/browse/NCF-47
D. NOT YET FIXED
Correctly read scalar characters over.
DAP non-transparency: Works locally, fails through DAP server.
Problem, IMHO, is with DAP definition/protocol
Demonstration:
ncks -O -D 1 -H -C -m --md5 -v md5_a -p
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods in.nc
20120801: Verified problem still exists
Bug report not filed
Cause: DAP translates scalar characters into 64-element, NUL-terminated
strings so MD5 agreement fails
"Sticky" reminders:
A. Pre-built, up-to-date Debian Sid & Ubuntu packages:
http://nco.sf.net#debian
B. Pre-built Fedora and CentOS RPMs:
http://nco.sf.net#rpm
C. Pre-built Windows (native) and Cygwin binaries:
http://nco.sf.net#windows
D. Pre-built AIX binaries:
http://nco.sf.net#aix
E. Did you try SWAMP (Script Workflow Analysis for MultiProcessing)?
SWAMP efficiently schedules/executes NCO scripts on remote servers:
http://swamp.googlecode.com
SWAMP can work command-line operator analysis scripts besides NCO.
If you must transfer lots of data from a server to your client
before you analyze it, then SWAMP will likely speed things up.
F. NCO support for netCDF4 features is tracked at
http://nco.sf.net/nco.html#nco4
NCO supports netCDF4 atomic data types, compression, and chunking.
G. Have you seen the NCO logo candidates by Tony Freeman, Rich
Signell, Rob Hetland, and Andrea Cimatoribus?
http://nco.sf.net
Tell us what you think...
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine (949) 891-2429 :)
