[netcdfgroup] netCDF Operators NCO version 4.6.3 are ready

The netCDF Operators NCO version 4.6.3 are ready.

http://nco.sf.net (Homepage, Mailing lists)
http://github.com/nco (Source Code, Releases, Developers)

What's new?

4.6.3 adds many new convenience features to existing functionality
like JSON, ncap2, ncremap, and ncclimo. Multi-dimensional bracketing
completes our JSON implementation. ncap2 adds a convenient UDUnits
conversion function. ncremap and ncclimo support long options.
ncclimo supports binary climatology generation and annual-mean mode.

Work on NCO 4.6.4 has commenced. Planned improvements include CMake
builds, more flexibility in handling extensive variables during
regridding.

Enjoy,
Charlie

NEW FEATURES (full details always in ChangeLog):

A. ncclimo supports "binary climos" and annual-mean mode.
   Binary climos are climos created from merging two climos, rather
   than re-computing a climos from raw input.
   This saves disk space and time for long climos.
   Annual-mean mode allows ncclimo to process input files that are
   annual rather than monthly means.
   http://nco.sf.net/nco.html#ncclimo

B. ncrcat and ncra now re-base data (move to a common time origin)
   from arbitrary time units in multiple calendar systems.
   Previously, re-basing only worked when the basetime (i.e., the
   YYMMDD in units like "XXX since YYMMDD") changed.
   Now rebasing takes into account the full units, both the increment
   (XXX) and the basetime (YYMMDD). Thanks to Dave Allured for the
   suggestion and Henry Butowsky for the re-implementation.
   http://nco.sf.net/nco.html#rbs

C. ncap2 now supports converting data between any two compatible units
   systems supported by UDUnits. The udunits() function takes and
   input variable and a UDUnits dimension string.
   T[lon]={0.0,100.0,150.0,200.0};
   T@units="Celsius";
   T=udunits(T,"kelvin");
   print(T);
   273.15, 373.15, 423.15, 473.15 ;
   The method auto-magically reads var_in@units and var_in@calendar
   (so, YES, this works with dates) attributes as necessary.
   Thanks to Henry Butowsky for this feature.
   http://nco.sf.net/nco.html#udunits_fnc

D. ncclimo and ncremap now support long options, e.g.,
   ncclimo --case=caseid --input=drc_in --output=drc_out --map=rgr_map
   ncremap --input_file=in_fl --destination=dst_fl --output_file=out_fl
   http://nco.sf.net/nco.html#ncclimo
   http://nco.sf.net/nco.html#ncremap

E. ncclimo and ncremap now save the full command line with which they
   were invoked as a single global attribute.
   Previously portions were saved as separate attributes.
   The new attributes are climo_command and remap_command.
   Their contents will exactly replicate (except for datestamps)
   the climatologies or regridded files they were created for.
   This improved provenance comes at the cost of up to a few kB more
   metadata in each file.

F. ncks can now print attribute CDL types as comments.
   CDL attribute types can be hard for humans to discern, so now
   ncks will print the type when invoked with -D 1 or greater.
   The printed file is fully CDL-compliant and works with ncgen.
   Credit to whomever first thought of this feature, and implemented
   it in some CDL output someone sent me.
   zender@firn:~/nco$ ncks -D 1 -C --cdl ~/nco/data/in_4.nc
   ...
   att_var:byte_att = 0b, 1b, 2b, 127b, -128b, -127b, -2b, -1b ; // byte
   att_var:char_att = "Sentence one.\n",
      "Sentence two.\n" ; // char
   att_var:short_att = 37s ; // short
   att_var:int_att = 73 ; // int
att_var:float_att = 73.f, 72.f, 71.f, 70.01f, 69.001f, 68.01f, 67.01f ; // float att_var:double_att = 73., 72., 71., 70.01, 69.001, 68.01, 67.010001 ; // double att_var:ubyte_att = 0ub, 1ub, 2ub, 127ub, 128ub, 254ub, 255ub, 0ub ; // ubyte
   att_var:ushort_att = 37us ; // ushort
   att_var:uint_att = 73ul ; // uint

G. JSON brackets
   Similar the CDL and XML backends, ncks supports JSON (as of 4.6.2).
   ncks now prints strided brackets to demarcate inner dimensions
   of multi-dimensional variable data.
   Invoking with --json vs. --jsn_fmt=4 on foo(2,3,4) yields:
   "data": [[[0.0, 1.0, 2.0, 3.0], [4.0, 5.0, 6.0, 7.0], [8.0, 9.0,
   10.0, 11.0]], [[12.0, 13.0, 14.0, 15.0], [16.0, 17.0, 18.0, 19.0],
   [20.0, 21.0, 22.0, 23.0]]]
   "data": [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0,
   11.0, 12.0, 13.0, 14.0, 15.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0,
   22.0, 23.0]
   Bracketed data are suitable for pasting into Python.
   More sample output at:
   http://dust.ess.uci.edu/tmp/in.json and other *.json files.
   Thanks to Henry Butowsky for implementing the bracketing.
   http://nco.sf.net/nco.html#json

BUG FIXES:

A. None!

KNOWN PROBLEMS DUE TO NCO:

   This section of ANNOUNCE reports and reminds users of the
   existence and severity of known, not yet fixed, problems.
   These problems occur with NCO 4.6.3 built/tested under
   MacOS 10.12.1 with netCDF 4.4.1 on HDF5 1.8.16 and with
   Linux with netCDF 4.4.2-development (20161116) on HDF5 1.8.16.

A. NOT YET FIXED (NCO problem)
Correctly read arrays of NC_STRING with embedded delimiters in ncatted arguments

   Demonstration:
ncatted -D 5 -O -a new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc ~/foo.nc
   ncks -m -C -v att_var ~/foo.nc

   20130724: Verified problem still exists
   TODO nco1102
   Cause: NCO parsing of ncatted arguments is not sophisticated
   enough to handle arrays of NC_STRINGS with embedded delimiters.

B. NOT YET FIXED (NCO problem?)
ncra/ncrcat (not ncks) hyperslabbing can fail on variables with multiple record dimensions

   Demonstration:
   ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc

   20140826: Verified problem still exists
   20140619: Problem reported by rmla
   Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
   Workaround: Convert to fixed dimensions then hyperslab

KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:

A. NOT YET FIXED (netCDF4 or HDF5 problem?)
   Specifying strided hyperslab on large netCDF4 datasets leads
   to slowdown or failure with recent netCDF versions.

   Demonstration with NCO <= 4.4.5:
   time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
   Demonstration with NCL:
   time ncl < ~/nco/data/ncl.ncl
   20140718: Problem reported by Parker Norton
   20140826: Verified problem still exists
   20140930: Finish NCO workaround for problem
   Cause: Slow algorithm in nc_var_gets()?
   Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
   Workaround #2: Convert file to netCDF3 first, then use stride

B. NOT YET FIXED (netCDF4 library bug)
Simultaneously renaming multiple dimensions in netCDF4 file can corrupt output

   Demonstration:
ncrename -O -d lev,z -d lat,y -d lon,x ~/nco/data/in_grp.nc ~/foo.nc # Completes but file is unreadable
   ncks -v one ~/foo.nc

20150922: Confirmed problem reported by Isabelle Dast, reported to Unidata
   20150924: Unidata confirmed problem
   20160212: Verified problem still exists in netCDF library
   20160512: Ditto
   20161028: Verified problem still exists with netCDF 4.4.1

   Bug tracking: https://www.unidata.ucar.edu/jira/browse/fxm
   More details: http://nco.sf.net/nco.html#ncrename_crd

C. FIXED in netCDF Development branch as of 20161116 and in maintenance release 4.4.1.1 nc-config/nf-config produce erroneous switches that cause NCO builds to fail
   This problem affects netCDF 4.4.1 on all operating systems.
   Some pre-compiled netCDF packages may have patched the problem.
   Hence it does not affect my MacPorts install of netCDF 4.4.1.

   Demonstration:
   % nc-config --cflags # Produces extraneous text that confuses make
   Using nf-config: /usr/local/bin/nf-config
   -I/usr/local/include -I/usr/local/include -I/usr/include/hdf

   If your nc-config output contains the "Using ..." line, you are
   affected by this issue.

   20161029: Reported problem to Unidata
20161101: Unidata confirmed reproducibility, attributed to netCDF 4.4.1 changes
   20161116: Unidata patch is in tree for netCDF 4.4.2 release
   20161123: Fixed in maintenance release netCDF 4.4.1.1

D. NOT YET FIXED (would require DAP protocol change?)
   Unable to retrieve contents of variables including period '.' in name
   Periods are legal characters in netCDF variable names.
   Metadata are returned successfully, data are not.
   DAP non-transparency: Works locally, fails through DAP server.

   Demonstration:
ncks -O -C -D 3 -v var_nm.dot -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to find variable

   20130724: Verified problem still exists.
   Stopped testing because inclusion of var_nm.dot broke all test scripts.
NB: Hard to fix since DAP interprets '.' as structure delimiter in HTTP query string.

   Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47

E. NOT YET FIXED (would require DAP protocol change)
   Correctly read scalar characters over DAP.
   DAP non-transparency: Works locally, fails through DAP server.
   Problem, IMHO, is with DAP definition/protocol

   Demonstration:
ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc

   20120801: Verified problem still exists
   Bug report not filed
   Cause: DAP translates scalar characters into 64-element (this
   dimension is user-configurable, but still...), NUL-terminated
   strings so MD5 agreement fails

"Sticky" reminders:

A. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g.,
   HDF4: AMSR MERRA MODIS ...
   HDF5: GLAS ICESat Mabel SBUV ...
   HDF-EOS5: AURA HIRDLS OMI ...

B. Pre-built executables for many OS's at:
   http://nco.sf.net#bnr
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(