[netcdfgroup] netCDF Operators NCO version 4.4.9 are ready

The netCDF Operators NCO version 4.4.9 are ready.

http://nco.sf.net (Homepage)
http://dust.ess.uci.edu/nco (Homepage "mirror")

This release includes small feature improvements in chunking,
deflation, and attribute handling during dimension reduction.
Also improvements in clarity and consistency of messages.
While chunking is an esoteric feature for many, its importance
is increasing as the move to netCDF4 (classic) continues.
The chunking defaults and flexibility have reached a new level
of robustness in this release.

The main new awesome feature is that ncra understands weights.
Finally. Applying uneven weights to different input files was
too clunky. This release is dedicated to the intrepid souls who
use NCO to generate climatologies that require uneven weights.
This release should make that task much easier.

Work on NCO 4.5.0 has commenced and will support regridding,
additional ncra flexibility with weights, and easier builds on
supercomputer systems that use modules.

Enjoy,
Charlie

NEW FEATURES (full details always in ChangeLog):

A. NCO has improved default performance of deflation and chunking.
   Our "best practices" chunking (map/policy=nco) is now default when
   manually deflating netCDF3 files into netCDF4 files.
   This invokes Rew's balanced chunking for three-dimensional
   variables, and LeFter-Product (cnk_map=lfp) chunking for others.
   LFP chunking, in turn, implements more reasonable defaults when
   the variable size is smaller than a single chunksize.
   This reduces excessive chunk sizes for many small variables.
   Shuffle is now turned-on by default when manually deflating files.
   This restores behavior from the NCO 4.3.x series, and can improve
   compression ratios by 10-30% relative to not shuffling.
   Copying multi-dimensional record variables from netCDF3 to netCDF4
   files no longer invokes the MM3 workaround.
   This speeds-up deflation of netCDF3 datasets.
   ncks -O -4 -L 1 netCDF3.nc netCDF4.nc
   http://nco.sf.net/nco.html#cnk

B. ncwa now eliminates dimensions from the "coordinates" attribute
   after collapsing them. This helps ensure that rank-reduced
   variables become completely independent from their former
   dimensions. The former presence of collapsed dimensions continues
   to be indicated by the "cell_methods" attribute.
   ncwa -O -a lat in.nc out.nc
   http://nco.sf.net/nco.html#coordinates
   http://nco.sf.net/nco.html#cell_methods
   http://nco.sf.net/nco.html#ncwa

C. NCO now warns users and suggest workarounds in situations when
   operators behave correctly, though perhaps not as expected,
   including:
   1. When coordinate variables intended to be excluded by -x and -C
      may nonetheless appear in output file, and suggest workaround
   2. When hyperslabs are specified for multi-dimensional
      "coordinates"

D. NCO now respects the CF "climatology" attribute: The variable
   pointed to by this attribute is treated as a pseudo-coordinate
   variable, and is extracted by default with any referring variable.
   Climatology variables now obey the same arithmetic rules as
   coordinates and are exempted from certain operations.
   Their treatment is identical to that of "bounds" variables.
   This command now retrieves the named variable, its coordinates (if
   any), and any variables named in the "ancillary_variable",
   "bounds", "climatology", and "coordinates" attributes:
   ncks -v temperature in.nc out.nc
   http://nco.sf.net/nco.html#climatology

E. ncra now accepts user-specified weights with the -w switch.
   When no weight is specified, ncra continues its old behavior
   and weights each record (e.g., time slice) equally.
   Weights specified with -w wgt may take one of two forms.
   Fist, the wgt argument may be a comma-separated list of values by
   which to weight each input file. Or, the wgt argument may be
   the name of a weighting variable present in every input file.
   The variable may be a scalar or a one-dimensional record variable.
   Scalar weights are applied uniformly to the entire file (i.e., a
   per-file weight), while one-dimensional weights apply to each
   corresponding record (i.e., per-record weights). Two applications
   of weights with ncra include easier generation of accurate averages
   of seasonal statistics, and of accurate averages of quantities
   sampled with a dynamically changing timestep.
   ncra -w 31,31,28 dec.nc jan.nc feb.nc out.nc
   ncra -w delta_tm in1.nc in2.nc in3.nc out.nc
   http://nco.sf.net/nco.html#ncra

BUG FIXES:

A. Fix ncecat in group aggregation mode (i.e., ncecat --gag) so it
   can no longer use uninitialized file pointers.

KNOWN PROBLEMS DUE TO NCO:

   This section of ANNOUNCE reports and reminds users of the
   existence and severity of known, not yet fixed, problems.
   These problems occur with NCO 4.4.9 built/tested with netCDF
   4.3.3-rc3 (20150129) on top of HDF5 hdf5-1.8.13 with:

   cd ~/nco;./configure # Configure mechanism -or-
   cd ~/nco/bld;make dir;make allinone # Old Makefile mechanism

A. NOT YET FIXED (NCO problem)
Correctly read arrays of NC_STRING with embedded delimiters in ncatted arguments

   Demonstration:
ncatted -D 5 -O -a new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc ~/foo.nc
   ncks -m -C -v att_var ~/foo.nc

   20130724: Verified problem still exists
   TODO nco1102
   Cause: NCO parsing of ncatted arguments is not sophisticated
   enough to handle arrays of NC_STRINGS with embedded delimiters.

B. NOT YET FIXED (NCO problem?)
ncra/ncrcat (not ncks) hyperslabbing can fail on variables with multiple record dimensions

   Demonstration:
   ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc

   20140826: Verified problem still exists
   20140619: Problem reported by rmla
   Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
   Workaround: Convert to fixed dimensions then hyperslab

KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:

A. NOT YET FIXED (netCDF4 or HDF5 problem?)
   Specifying strided hyperslab on large netCDF4 datasets leads
   to slowdown or failure with recent netCDF versions.

   Demonstration with NCO <= 4.4.5:
   time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
   Demonstration with NCL:
   time ncl < ~/nco/data/ncl.ncl
   20140718: Problem reported by Parker Norton
   20140826: Verified problem still exists
   20140930: Finish NCO workaround for problem
   Cause: Slow algorithm in nc_var_gets()?
   Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
   Workaround #2: Convert file to netCDF3 first, then use stride

B. NOT YET FIXED (would require DAP protocol change?)
   Unable to retrieve contents of variables including period '.' in name
   Periods are legal characters in netCDF variable names.
   Metadata are returned successfully, data are not.
   DAP non-transparency: Works locally, fails through DAP server.

   Demonstration:
ncks -O -C -D 3 -v var_nm.dot -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to find variable

   20130724: Verified problem still exists.
   Stopped testing because inclusion of var_nm.dot broke all test scripts.
NB: Hard to fix since DAP interprets '.' as structure delimiter in HTTP query string.

   Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47

C. NOT YET FIXED (would require DAP protocol change)
   Correctly read scalar characters over DAP.
   DAP non-transparency: Works locally, fails through DAP server.
   Problem, IMHO, is with DAP definition/protocol

   Demonstration:
ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc

   20120801: Verified problem still exists
   Bug report not filed
   Cause: DAP translates scalar characters into 64-element (this
   dimension is user-configurable, but still...), NUL-terminated
   strings so MD5 agreement fails

"Sticky" reminders:

A. Pre-built Debian Sid & Ubuntu packages:
   http://nco.sf.net#debian

B. Pre-built Fedora and CentOS RPMs:
   http://nco.sf.net#rpm

C. Pre-built Mac binaries:
   http://nco.sf.net#mac

D. Pre-built Windows (native) and Cygwin binaries:
   http://nco.sf.net#windows

E. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g.,
   HDF4: AMSR MERRA MODIS ...
   HDF5: GLAS ICESat Mabel SBUV ...
   HDF-EOS5: AURA HIRDLS OMI ...

--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(



  • 2015 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: