The netCDF Operators NCO version 4.3.5 are ready.
http://nco.sf.net (Homepage)
http://dust.ess.uci.edu/nco (Homepage "mirror")
The current release brings group support to ncra, ncea, and ncrcat.
Now ten operators fully support group hierarchies: ncbo, ncea, ncecat,
ncflint, ncks, ncpdq, ncra, ncrcat, ncrename, and ncwa. The operators
that do no yet fully support group hierarchies are ncatted and ncap2.
Rounding errors can accumulate to worrisome levels during arithmetic
performed on large (>~10,000) arrays of single precision floats.
Now NCO has the --dbl switch that forces operators to coerce
single-precision numbers to double-precision for arithemetic.
It is expensive in terms of time and RAM, and often not warranted.
Use this switch to guarantee preserving maximum precision.
Or if you want your NCO answers on single-precision fields to always
agree exactly with tools that always promote (NCL, Matlab, IDL).
The manual now contains a detailed discussion of the issues.
Unfortunately progress sometimes comes at a price in bugs :)
4.3.5 fixes a nasty ncwa bug present in 4.3.3 and 4.3.4.
Nasty means that it could produce wrong answers that look right.
Fortunately, it could only occur when simultaneously averaging and
hyperslabbing, e.g., ncwa -a lat,lon -d lat,0.,90. in.nc out.nc.
Such users should immediately upgrade to 4.3.5 and re-check results.
Full details below.
Work on NCO 4.3.6 is underway and includes improved netCDF4 support
for more NCO operators (ncatted) and improved support for HDF4 files.
Enjoy,
Charlie
"New stuff" in 4.3.5 summary (full details always in ChangeLog):
NEW FEATURES:
A. ncra, ncea, and ncrcat now supports groups, Group Path Editing
(GPE) and list unions, the same options already supported by ncbo,
ncecat, ncflint, ncks, ncpdq, and ncwa.
Features are controlled by -g, -G and --union, respectively.
ncra -g g1 -v v1 -G dude -O -p ~/nco/data in_grp.nc ~/foo.nc
ncea -g g1 -v v1 -G dude -O -p ~/nco/data in_grp.nc ~/foo.nc
ncrcat -g g1 -v v1 -G dude -O -p ~/nco/data in_grp.nc ~/foo.nc
http://nco.sf.net/nco.html#ncea
http://nco.sf.net/nco.html#ncra
http://nco.sf.net/nco.html#ncrcat
http://nco.sf.net/nco.html#grp
http://nco.sf.net/nco.html#gpe
http://nco.sf.net/nco.html#union
B. Forced conversion of float to double:
NCO has always converted integers and single precision reals (i.e.,
floats) to double precision reals (i.e., doubles) prior to
arithmetic when it would preserve precision of the result (the
results are converted back to the original type after arithmetic).
Until now, NCO never implicitly converted floats to doubles unless
doubles were involved in the computation, because, well, GIGO
("garbage in, garbage out") and wisdom from Numerical Recipes.
However, such promotion is often warranted on arithmetic involving
large (> ~10,000) numbers of single-precision inputs.
Now NCO allows a new arithmetic convention to implement this.
The convention is manually invoked with --dbl:
ncea --dbl in*.nc out.nc
ncra --dbl in*.nc out.nc
ncwa --dbl in.nc out.nc
The current default behavior is explicitly selected with --flt.
Depending on user feedback, --dbl may become the default in
future versions of NCO. Send me your opinion!
NB: Invoking arithmetic promotion of NC_FLOAT to NC_DOUBLE
increases RAM memory usage by a factor of two, and slows down
arithmetic, though perhaps negligibly compared to I/O time.
The manual contains a detailed discussion of the issues:
http://nco.sf.net/nco.html#dbl
C. CDL-mode prints more verbose information as comments (after '// ')
as the debugging level increases (i.e., ncks --cdl -D 1).
Currently this includes a file summary, the ncgen command required
to convert back to netCDF, and full path names for all groups.
ncks --cdl -D 1 ~/nco/data/in_grp.nc # Adds file summary
ncks --cdl -D 3 ~/nco/data/in_grp.nc # Adds full groupnames
http://nco.sf.net/nco.html#cdl
D. GPE operations allow netCDF3 output from netCDF4 input when groups
are not required in the output file. This allows GPE flattening to
directly create netCDF3 classic files for backward compatibility.
For example, this extracts group "cesm" from a netCDF4 file and
stores it at the root level of a netCDF3 output file:
ncks -O -G : -g cesm -3 ~/nco/data/cmip5.nc ~/cesm.nc
http://nco.sf.net/nco.html#gpe
http://nco.sf.net/nco.html#xmp_flt
BUG FIXES:
A. ncwa hyperslabbing while averaging bug:
Versions 4.3.3-4.3.4 of ncwa could return incorrect answers when
user-specified hyperslabs were simultaneously extracted.
In such cases, hyperslab limits were not consistently applied.
This could produce incorrect answers that look correct.
This bug only affected hyperslabbed statistics (those produced
by simultaneously invoking -a and -d switches);
"global averages" were unaffected.
We urge all ncwa users to upgrade to NCO 4.3.5.
http://nco.sf.net#bug_ncwa_lmt
B. Group operators now correctly match variable names that are
proper subsets of ancestor group names.
C. Fixed ncpdq dimension reversal in group hierarchies.
KNOWN ISSUES NOT YET FIXED:
This section of ANNOUNCE reports and reminds users of the
existence and severity of known, not yet fixed, problems.
These problems occur with NCO 4.3.5 built/tested with netCDF
4.3.1-rc3 snapshot 20130827 on top of HDF5 hdf5-1.8.9 with these
methods:
cd ~/nco;./configure --enable-netcdf4 # Configure mechanism -or-
cd ~/nco/bld;make dir;make all;make ncap2 # Old Makefile mechanism
A. NOT YET FIXED
Correctly read netCDF4 input over DAP, write netCDF4 output, then
read resulting file.
Replacing netCDF4 with netCDF3 in either location of preceding
sentence leads to success.
DAP non-transparency: Works locally, fails through DAP server.
Unclear whether resulting file is "legal" because of dimension ID
ordering assumptions.
Demonstration:
ncks -4 -O -v three_dmn_rec_var
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods/in_4.nc ~/foo.nc
ncks ~/foo.nc # breaks with "NetCDF: Invalid dimension ID or name"
20130724: Unable to verify since in_4.nc no longer accessible on
Unidata DAP server
Bug report filed: netCDF #QUN-641037: dimension ID ordering assumptions
B. NOT YET FIXED
netCDF4 library fails when renaming dimension and variable using
that dimension, in either order. Works fine with netCDF3.
Problem with netCDF4 library implementation.
Demonstration:
ncks -O -4 -v lat_T42 ~/nco/data/in.nc ~/foo.nc
ncrename -O -D 2 -d lat_T42,lat -v lat_T42,lat ~/foo.nc ~/foo2.nc #
Breaks with "NetCDF: HDF error"
ncks -m ~/foo.nc
20130724: Verified problem still exists
Bug report filed: netCDF #YQN-334036: problem renaming dimension and
coordinate in netCDF4 file
C. NOT YET FIXED (requires change to DAP protocol)
Unable to retrieve contents of variables including period '.' in name
Periods are legal characters in netCDF variable names.
Metadata are returned successfully, data are not.
DAP non-transparency: Works locally, fails through DAP server.
Demonstration:
ncks -O -C -D 3 -v var_nm.dot -p
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods in.nc # Fails to
find variable
20130724: Verified problem still exists.
Stopped testing because inclusion of var_nm.dot broke all test scripts.
NB: Hard to fix since DAP interprets '.' as structure delimiter in
HTTP query string.
Bug report filed: https://www.unidata.ucar.edu/jira/browse/NCF-47
D. NOT YET FIXED (requires change to DAP protocol)
Correctly read scalar characters over DAP.
DAP non-transparency: Works locally, fails through DAP server.
Problem, IMHO, is with DAP definition/protocol
Demonstration:
ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods in.nc
20120801: Verified problem still exists
Bug report not filed
Cause: DAP translates scalar characters into 64-element (this
dimension is user-configurable, but still...), NUL-terminated
strings so MD5 agreement fails
E. NOT YET FIXED (NCO problem)
Correctly read arrays of NC_STRING with embedded delimiters in
ncatted arguments
Demonstration:
ncatted -D 5 -O -a
new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc
~/foo.nc
20130724: Verified problem still exists
TODO nco1102
Cause: NCO parsing of ncatted arguments is not yet sophisticated
enough to handle arrays of NC_STRINGS with embedded delimiters.
"Sticky" reminders:
A. Pre-built, up-to-date Debian Sid & Ubuntu packages:
http://nco.sf.net#debian
B. Pre-built Fedora and CentOS RPMs:
http://nco.sf.net#rpm
C. Pre-built Windows (native) and Cygwin binaries:
http://nco.sf.net#windows
D. Pre-built AIX binaries:
http://nco.sf.net#aix
E. Did you try SWAMP (Script Workflow Analysis for MultiProcessing)?
SWAMP efficiently schedules/executes NCO scripts on remote servers:
http://swamp.googlecode.com
SWAMP can work command-line operator analysis scripts besides NCO.
If you must transfer lots of data from a server to your client
before you analyze it, then SWAMP will likely speed things up.
F. NCO support for netCDF4 features is tracked at
http://nco.sf.net/nco.html#nco4
NCO supports netCDF4 atomic data types, compression, chunking, and
groups.
G. Reminder that ncks, ncecat, ncbo, ncflint, and ncpdq work on many common
HDF5 datasets, e.g.,
NASA AURA HIRDLS HDF-EOS5
NASA ICESat GLAS HDF5
NASA SBUV HDF5...
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(
