[netcdfgroup] netCDF Operators NCO version 4.3.5 are ready

The netCDF Operators NCO version 4.3.5 are ready.

http://nco.sf.net (Homepage)
http://dust.ess.uci.edu/nco (Homepage "mirror")

The current release brings group support to ncra, ncea, and ncrcat.
Now ten operators fully support group hierarchies: ncbo, ncea, ncecat,
ncflint, ncks, ncpdq, ncra, ncrcat, ncrename, and ncwa. The operators
that do no yet fully support group hierarchies are ncatted and ncap2.

Rounding errors can accumulate to worrisome levels during arithmetic
performed on large (>~10,000) arrays of single precision floats.
Now NCO has the --dbl switch that forces operators to coerce
single-precision numbers to double-precision for arithemetic.
It is expensive in terms of time and RAM, and often not warranted.
Use this switch to guarantee preserving maximum precision.
Or if you want your NCO answers on single-precision fields to always
agree exactly with tools that always promote (NCL, Matlab, IDL).
The manual now contains a detailed discussion of the issues.

Unfortunately progress sometimes comes at a price in bugs :)
4.3.5 fixes a nasty ncwa bug present in 4.3.3 and 4.3.4.
Nasty means that it could produce wrong answers that look right.
Fortunately, it could only occur when simultaneously averaging and
hyperslabbing, e.g., ncwa -a lat,lon -d lat,0.,90. in.nc out.nc.
Such users should immediately upgrade to 4.3.5 and re-check results.
Full details below.

Work on NCO 4.3.6 is underway and includes improved netCDF4 support
for more NCO operators (ncatted) and improved support for HDF4 files.

Enjoy,
Charlie

"New stuff" in 4.3.5 summary (full details always in ChangeLog):

NEW FEATURES:

A. ncra, ncea, and ncrcat now supports groups, Group Path Editing
   (GPE) and list unions, the same options already supported by ncbo,
   ncecat, ncflint, ncks, ncpdq, and ncwa.
   Features are controlled by -g, -G and --union, respectively.
   ncra -g g1 -v v1 -G dude -O -p ~/nco/data in_grp.nc ~/foo.nc
   ncea -g g1 -v v1 -G dude -O -p ~/nco/data in_grp.nc ~/foo.nc
   ncrcat -g g1 -v v1 -G dude -O -p ~/nco/data in_grp.nc ~/foo.nc
   http://nco.sf.net/nco.html#ncea
   http://nco.sf.net/nco.html#ncra
   http://nco.sf.net/nco.html#ncrcat
   http://nco.sf.net/nco.html#grp
   http://nco.sf.net/nco.html#gpe
   http://nco.sf.net/nco.html#union

B. Forced conversion of float to double:
   NCO has always converted integers and single precision reals (i.e.,
   floats) to double precision reals (i.e., doubles) prior to
   arithmetic when it would preserve precision of the result (the
   results are converted back to the original type after arithmetic).
   Until now, NCO never implicitly converted floats to doubles unless
   doubles were involved in the computation, because, well, GIGO
   ("garbage in, garbage out") and wisdom from Numerical Recipes.
   However, such promotion is often warranted on arithmetic involving
   large (> ~10,000) numbers of single-precision inputs.
   Now NCO allows a new arithmetic convention to implement this.
   The convention is manually invoked with --dbl:
   ncea --dbl in*.nc out.nc
   ncra --dbl in*.nc out.nc
   ncwa --dbl in.nc  out.nc
   The current default behavior is explicitly selected with --flt.
   Depending on user feedback, --dbl may become the default in
   future versions of NCO. Send me your opinion!
   NB: Invoking arithmetic promotion of NC_FLOAT to NC_DOUBLE
   increases RAM memory usage by a factor of two, and slows down
   arithmetic, though perhaps negligibly compared to I/O time.
   The manual contains a detailed discussion of the issues:
   http://nco.sf.net/nco.html#dbl

C. CDL-mode prints more verbose information as comments (after '// ')
   as the debugging level increases (i.e., ncks --cdl -D 1).
   Currently this includes a file summary, the ncgen command required
   to convert back to netCDF, and full path names for all groups.
   ncks --cdl -D 1 ~/nco/data/in_grp.nc # Adds file summary
   ncks --cdl -D 3 ~/nco/data/in_grp.nc # Adds full groupnames
   http://nco.sf.net/nco.html#cdl

D. GPE operations allow netCDF3 output from netCDF4 input when groups
   are not required in the output file. This allows GPE flattening to
   directly create netCDF3 classic files for backward compatibility.
   For example, this extracts group "cesm" from a netCDF4 file and
   stores it at the root level of a netCDF3 output file:
   ncks -O -G : -g cesm -3 ~/nco/data/cmip5.nc ~/cesm.nc
   http://nco.sf.net/nco.html#gpe
   http://nco.sf.net/nco.html#xmp_flt

BUG FIXES:

A. ncwa hyperslabbing while averaging bug:
   Versions 4.3.3-4.3.4 of ncwa could return incorrect answers when
   user-specified hyperslabs were simultaneously extracted.
   In such cases, hyperslab limits were not consistently applied.
   This could produce incorrect answers that look correct.
   This bug only affected hyperslabbed statistics (those produced
   by simultaneously invoking -a and -d switches);
   "global averages" were unaffected.
   We urge all ncwa users to upgrade to NCO 4.3.5.
   http://nco.sf.net#bug_ncwa_lmt

B. Group operators now correctly match variable names that are
   proper subsets of ancestor group names.

C. Fixed ncpdq dimension reversal in group hierarchies.

KNOWN ISSUES NOT YET FIXED:

   This section of ANNOUNCE reports and reminds users of the
   existence and severity of known, not yet fixed, problems.
   These problems occur with NCO 4.3.5 built/tested with netCDF
   4.3.1-rc3 snapshot 20130827 on top of HDF5 hdf5-1.8.9 with these
methods:

   cd ~/nco;./configure --enable-netcdf4  # Configure mechanism -or-
   cd ~/nco/bld;make dir;make all;make ncap2 # Old Makefile mechanism

A. NOT YET FIXED
   Correctly read netCDF4 input over DAP, write netCDF4 output, then
read resulting file.
   Replacing netCDF4 with netCDF3 in either location of preceding
sentence leads to success.
   DAP non-transparency: Works locally, fails through DAP server.
   Unclear whether resulting file is "legal" because of dimension ID
ordering assumptions.

   Demonstration:
   ncks -4 -O -v three_dmn_rec_var
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods/in_4.nc ~/foo.nc
   ncks ~/foo.nc # breaks with "NetCDF: Invalid dimension ID or name"

   20130724: Unable to verify since in_4.nc no longer accessible on
Unidata DAP server
   Bug report filed: netCDF #QUN-641037: dimension ID ordering assumptions

B. NOT YET FIXED
   netCDF4 library fails when renaming dimension and variable using
   that dimension, in either order. Works fine with netCDF3.
   Problem with netCDF4 library implementation.

   Demonstration:
   ncks -O -4 -v lat_T42 ~/nco/data/in.nc ~/foo.nc
   ncrename -O -D 2 -d lat_T42,lat -v lat_T42,lat ~/foo.nc ~/foo2.nc #
Breaks with "NetCDF: HDF error"
   ncks -m ~/foo.nc

   20130724: Verified problem still exists
   Bug report filed: netCDF #YQN-334036: problem renaming dimension and
coordinate in netCDF4 file

C. NOT YET FIXED (requires change to DAP protocol)
   Unable to retrieve contents of variables including period '.' in name
   Periods are legal characters in netCDF variable names.
   Metadata are returned successfully, data are not.
   DAP non-transparency: Works locally, fails through DAP server.

   Demonstration:
   ncks -O -C -D 3 -v var_nm.dot -p
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods in.nc # Fails to
find variable

   20130724: Verified problem still exists.
   Stopped testing because inclusion of var_nm.dot broke all test scripts.
   NB: Hard to fix since DAP interprets '.' as structure delimiter in
HTTP query string.

   Bug report filed: https://www.unidata.ucar.edu/jira/browse/NCF-47

D. NOT YET FIXED (requires change to DAP protocol)
   Correctly read scalar characters over DAP.
   DAP non-transparency: Works locally, fails through DAP server.
   Problem, IMHO, is with DAP definition/protocol

   Demonstration:
   ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods in.nc

   20120801: Verified problem still exists
   Bug report not filed
   Cause: DAP translates scalar characters into 64-element (this
   dimension is user-configurable, but still...), NUL-terminated
   strings so MD5 agreement fails

E. NOT YET FIXED (NCO problem)
   Correctly read arrays of NC_STRING with embedded delimiters in
ncatted arguments

   Demonstration:
   ncatted -D 5 -O -a
new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc
~/foo.nc

   20130724: Verified problem still exists
   TODO nco1102
   Cause: NCO parsing of ncatted arguments is not yet sophisticated
   enough to handle arrays of NC_STRINGS with embedded delimiters.

"Sticky" reminders:

A. Pre-built, up-to-date Debian Sid & Ubuntu packages:
   http://nco.sf.net#debian

B. Pre-built Fedora and CentOS RPMs:
   http://nco.sf.net#rpm

C. Pre-built Windows (native) and Cygwin binaries:
   http://nco.sf.net#windows

D. Pre-built AIX binaries:
   http://nco.sf.net#aix

E. Did you try SWAMP (Script Workflow Analysis for MultiProcessing)?
   SWAMP efficiently schedules/executes NCO scripts on remote servers:

   http://swamp.googlecode.com

   SWAMP can work command-line operator analysis scripts besides NCO.
   If you must transfer lots of data from a server to your client
   before you analyze it, then SWAMP will likely speed things up.

F. NCO support for netCDF4 features is tracked at

   http://nco.sf.net/nco.html#nco4

   NCO supports netCDF4 atomic data types, compression, chunking, and
groups.

G. Reminder that ncks, ncecat, ncbo, ncflint, and ncpdq work on many common
   HDF5 datasets, e.g.,
   NASA AURA HIRDLS HDF-EOS5
   NASA ICESat GLAS HDF5
   NASA SBUV HDF5...

-- 
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(



  • 2013 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: