The netCDF Operators NCO version 4.6.4 are ready.
http://nco.sf.net (Homepage, Mailing lists)
http://github.com/nco (Source Code, Releases, Developers)
What's new?
The main new features in 4.6.4 are the "splitter" and daily
climatology capabilities for ncclimo. ncclimo now generates,
combines existing, and splits climatologies. Thus it is a
fairly general purpose climatology operator. NCO now supports
two more CF conventions (formula_terms and cell_measures),
and has a few bugfixes.
Work on NCO 4.6.5 has commenced. Planned improvements include
support for conda installs on MS Windows, and more ncclimo
and ncremap features.
Enjoy,
Charlie
NEW FEATURES (full details always in ChangeLog):
A. ncclimo will reshape timeseries, i.e., split input datasets into
per-variable files that span the entire timeseries.
Input files are specified via stdin, via positional arguments
(after all options) on the command-line, or as the set of .nc files
in drc_in
ls $drc_in/*.nc | ncclimo -v T,Q --output=drc_out --map=rgr_map
ncclimo -v T,Q --output=drc_out --map=rgr_map < list
ncclimo -v T,Q --output=drc_out --map=rgr_map $drc_in/*.nc
ncclimo -v T,Q --input=$drc_in --output=drc_out --map=rgr_map
http://nco.sf.net/nco.html#ncclimo
B. ncks allows setting per-variable chunk cache-size with --cnk_csh.
The effects of varying chunk cache-size are not well understood,
at least by we who work on NCO. Users asked for a knob to tune
it, and now that exists. Experiment with this and if you find
reasons for NCO to set this to non-default values, let us know.
If you want more knobs, let us know.
This setting only works on netCDF4 files.
ncks -D 3 --cnk_csh 1000000 in.nc4 out.nc
http://nco.sf.net/nco.html#cnk_csh
C. ncremap now includes staggered grid weights with FV grid output.
The weights are named w_stag, following the CESM CAM convention.
http://nco.sf.net/nco.html#ncremap
D. Variables listed in a formula_terms attribute are now extracted
by default. Turn-off this behavior with --no_frm_trm.
The CF formula_terms convention is used to associate variables
with terms in defined formulas, such as vertical coordinates:
lev:standard_name = "atmosphere_hybrid_sigma_pressure_coordinate"
lev:formula_terms = "a: hyam b: hybm p0: P0 ps: PS"
Unless the default behavior is overridden, NCO attaches any
formula variables to the extraction list along with the primary
variable and other associated variables. By definition, formula
variables are a subset of the rank of the variable they define.
http://nco.sf.net/nco.html#frm_trm
E. Variables listed in a cell_measures attribute are now extracted
by default. Turn-off this behavior with --no_cll_msr.
This convention allows variables to indicate which other variables
contains area or volume information about a gridcell.
These measures variables are pointed to by the cell_measures
attribute. Unless the default behavior is overridden, NCO attaches
all measures variables to the extraction list along with the
primary variable.
http://nco.sf.net/nco.html#cll_msr
BUG FIXES:
A. Fix bug introduced in 4.6.3 where hyperslabs involving UDUnits
dates could fail, depending on exactly how NCO was compiled.
Failures most likely on MacOS because clang, unlike gcc, does not
initialize pointers to NULL by default. Solution is to upgrade.
A. Fix bug introduced in ~4.4.6 where some hyperslabs of non-unity
stride in netCDF4 files could return incorrect values.
If you use the hyperslab stride argument on netCDF4 files, we
suggest you upgrade immediately. The workaround is to convert
to netCDF3 before using a strided hyperslab.
Thanks to Martin Dix for reporting this bug.
KNOWN PROBLEMS DUE TO NCO:
This section of ANNOUNCE reports and reminds users of the
existence and severity of known, not yet fixed, problems.
These problems occur with NCO 4.6.4 built/tested under
MacOS 10.12.1 with netCDF 4.4.1 on HDF5 1.8.16 and with
Linux with netCDF 4.4.2-development (20161116) on HDF5 1.8.16.
A. NOT YET FIXED (NCO problem)
Correctly read arrays of NC_STRING with embedded delimiters in
ncatted arguments
Demonstration:
ncatted -D 5 -O -a
new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc
~/foo.nc
ncks -m -C -v att_var ~/foo.nc
20130724: Verified problem still exists
TODO nco1102
Cause: NCO parsing of ncatted arguments is not sophisticated
enough to handle arrays of NC_STRINGS with embedded delimiters.
B. NOT YET FIXED (NCO problem?)
ncra/ncrcat (not ncks) hyperslabbing can fail on variables with
multiple record dimensions
Demonstration:
ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc
20140826: Verified problem still exists
20140619: Problem reported by rmla
Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
Workaround: Convert to fixed dimensions then hyperslab
KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:
A. NOT YET FIXED (netCDF4 or HDF5 problem?)
Specifying strided hyperslab on large netCDF4 datasets leads
to slowdown or failure with recent netCDF versions.
Demonstration with NCO <= 4.4.5:
time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
Demonstration with NCL:
time ncl < ~/nco/data/ncl.ncl
20140718: Problem reported by Parker Norton
20140826: Verified problem still exists
20140930: Finish NCO workaround for problem
Cause: Slow algorithm in nc_var_gets()?
Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
Workaround #2: Convert file to netCDF3 first, then use stride
B. NOT YET FIXED (netCDF4 library bug)
Simultaneously renaming multiple dimensions in netCDF4 file can
corrupt output
Demonstration:
ncrename -O -d lev,z -d lat,y -d lon,x ~/nco/data/in_grp.nc ~/foo.nc
# Completes but file is unreadable
ncks -v one ~/foo.nc
20150922: Confirmed problem reported by Isabelle Dast, reported to
Unidata
20150924: Unidata confirmed problem
20160212: Verified problem still exists in netCDF library
20160512: Ditto
20161028: Verified problem still exists with netCDF 4.4.1
Bug tracking: https://www.unidata.ucar.edu/jira/browse/fxm
More details: http://nco.sf.net/nco.html#ncrename_crd
C. FIXED in netCDF Development branch as of 20161116 and in maintenance
release 4.4.1.1
nc-config/nf-config produce erroneous switches that cause NCO builds
to fail
This problem affects netCDF 4.4.1 on all operating systems.
Some pre-compiled netCDF packages may have patched the problem.
Hence it does not affect my MacPorts install of netCDF 4.4.1.
Demonstration:
% nc-config --cflags # Produces extraneous text that confuses make
Using nf-config: /usr/local/bin/nf-config
-I/usr/local/include -I/usr/local/include -I/usr/include/hdf
If your nc-config output contains the "Using ..." line, you are
affected by this issue.
20161029: Reported problem to Unidata
20161101: Unidata confirmed reproducibility, attributed to netCDF
4.4.1 changes
20161116: Unidata patch is in tree for netCDF 4.4.2 release
20161123: Fixed in maintenance release netCDF 4.4.1.1
D. NOT YET FIXED (would require DAP protocol change?)
Unable to retrieve contents of variables including period '.' in name
Periods are legal characters in netCDF variable names.
Metadata are returned successfully, data are not.
DAP non-transparency: Works locally, fails through DAP server.
Demonstration:
ncks -O -C -D 3 -v var_nm.dot -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to
find variable
20130724: Verified problem still exists.
Stopped testing because inclusion of var_nm.dot broke all test scripts.
NB: Hard to fix since DAP interprets '.' as structure delimiter in
HTTP query string.
Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47
E. NOT YET FIXED (would require DAP protocol change)
Correctly read scalar characters over DAP.
DAP non-transparency: Works locally, fails through DAP server.
Problem, IMHO, is with DAP definition/protocol
Demonstration:
ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc
20120801: Verified problem still exists
Bug report not filed
Cause: DAP translates scalar characters into 64-element (this
dimension is user-configurable, but still...), NUL-terminated
strings so MD5 agreement fails
"Sticky" reminders:
A. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g.,
HDF4: AMSR MERRA MODIS ...
HDF5: GLAS ICESat Mabel SBUV ...
HDF-EOS5: AURA HIRDLS OMI ...
B. Pre-built executables for many OS's at:
http://nco.sf.net#bnr
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(
