The netCDF Operators NCO version 4.6.2 are ready.
http://nco.sf.net (Homepage, Mailing lists)
http://github.com/nco (Source Code, Releases, Developers)
What's new?
4.6.2 is mainly a stability release to polish existing features and to
add minor new ones. The exception is the new JSON backend.
NCO now prints CDL, "Traditional NCO", XML, and JSON. Babel-icious eh?
We built the JSON backend to help a project (DOE Terraref).
Our first choice was to adopt an off-the-shelf netCDF->JSON tool.
However, no existing solution worked for us.
JSON is a loose syntax, and we made necessary design choices that
suited our application, and left some choices for later.
Are there syntactical variants you want us to add?
Some users of netCDF version 4.4.1 cannot build NCO from scratch
because a bug in the nc-config command kills NCO's 'configure;make'.
Unidata will ship a corrected nc-config in netCDF 4.4.2.
The 4.4.2-development branch already contains the necessary fix.
See (C) in KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS below.
Work on NCO 4.6.3 has commenced. Planned improvements include more
flexibility in handling extensive variables during regridding, CMake
support, and brackets for multi-dimensional array values in JSON.
Enjoy,
Charlie
NEW FEATURES (full details always in ChangeLog):
A. JSON backend
Similar the CDL and XML backends, ncks now supports JSON.
To obtain JSON, use --jsn or --json instead of --cdl/--xml.
ncks --json -v one ~/nco/data/in.nc
{
"variables": {
"one": {
"type": "float",
"attributes": {
"long_name": "one"
},
"data": 1.0
}
}
}
More sample output at:
http://dust.ess.uci.edu/tmp/in.json and other *.json files.
Other related options (-v, -g, -m, -M) have their usual meanings.
The --jsn_att_fmt switch accesses formats of selectable verbosity.
Thanks to Henry Butowsky for implementing the JSON backend,
and to Pedro Vicente and Chris Barker for helpful discussions.
http://nco.sf.net/nco.html#json
B. Multi-arguments refer to the format of command-line key-value pairs
that NCO uses to support four different features that can have too
many parameters to set via traditional command line arguments.
The four features are regridding (rgr), global-attribute-adding
(gaa), precision-preserving compression (ppc), and ENVI image
processing for the DOE Terraref project (trr). Formerly, users were
required to supply one value per command line option for these
features, e.g.,
"--rgr lat_nbr=64 --rgr lon_nbr=128 --rgr lat_grd=cap ..."
The new multi-argument feature allows users to aggregate multiple
key-value pairs per command line argument, e.g.,
"--rgr lat_nbr=64#lon_nbr=128#lat_grd=cap# ..."
Now all key-value pairs related to a single feature (like
regridding, compression) can be provided as one arument!
The string that delimits each pair is user-configurable and
defaults to a single hash "#". Change it with, e.g., --dlm=":".
Ensure your delimiters are not special shell characters, and are
escaped (with backslashes) if present in key or value strings.
Thanks to Jerome Mao for implementing multi-arguments.
http://nco.sf.net/nco.html#mta
C. ncap2 now searches all paths in the NCO_PATH environment variable
for any #include'd files that it cannot find in ${CWD}.
D. All builds tested on LLVM 8.0.0 and GCC 6.2.1 toolchains.
BUG FIXES:
A. ncclimo and ncremap could fail if ${BASH_SOURCE[0]} is unavailable.
This could occur, e.g., with the dash shell and/or due to software
modules that modify users' environments.
This portability issue has been resolved.
B. ncatted failed to implement NaN on some MinGW systems with, e.g.,
ncatted -O -a _FillValue,fll_val,m,f,nan ~/nco/data/in.nc ~/foo.nc
We think this has been fixed. The workaround is to upgrade or use
ncatted on *NIX or MacOS. Thanks to James Adams for reporting this.
C. ncks CDL now prints correct tokens for uint, int64, uint64 types.
These are "ul", "ll", "ull", respectively. Previously NCO used
different tokens because ncgen <= 4.3.0 choked on "ll" formats.
So this was an intentional NCO "feature", not a bug.
Use ncgen >= 4.3.1 to parse these CDL files.
KNOWN PROBLEMS DUE TO NCO:
This section of ANNOUNCE reports and reminds users of the
existence and severity of known, not yet fixed, problems.
These problems occur with NCO 4.6.2 built/tested under
MacOS 10.12.1 with netCDF 4.4.1 on HDF5 1.8.16 and with
Linux with netCDF 4.4.2-development (20161116) on HDF5 1.8.16.
A. NOT YET FIXED (NCO problem)
Correctly read arrays of NC_STRING with embedded delimiters in
ncatted arguments
Demonstration:
ncatted -D 5 -O -a
new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc
~/foo.nc
ncks -m -C -v att_var ~/foo.nc
20130724: Verified problem still exists
TODO nco1102
Cause: NCO parsing of ncatted arguments is not sophisticated
enough to handle arrays of NC_STRINGS with embedded delimiters.
B. NOT YET FIXED (NCO problem?)
ncra/ncrcat (not ncks) hyperslabbing can fail on variables with
multiple record dimensions
Demonstration:
ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc
20140826: Verified problem still exists
20140619: Problem reported by rmla
Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
Workaround: Convert to fixed dimensions then hyperslab
KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:
A. NOT YET FIXED (netCDF4 or HDF5 problem?)
Specifying strided hyperslab on large netCDF4 datasets leads
to slowdown or failure with recent netCDF versions.
Demonstration with NCO <= 4.4.5:
time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
Demonstration with NCL:
time ncl < ~/nco/data/ncl.ncl
20140718: Problem reported by Parker Norton
20140826: Verified problem still exists
20140930: Finish NCO workaround for problem
Cause: Slow algorithm in nc_var_gets()?
Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
Workaround #2: Convert file to netCDF3 first, then use stride
B. NOT YET FIXED (netCDF4 library bug)
Simultaneously renaming multiple dimensions in netCDF4 file can
corrupt output
Demonstration:
ncrename -O -d lev,z -d lat,y -d lon,x ~/nco/data/in_grp.nc ~/foo.nc
# Completes but file is unreadable
ncks -v one ~/foo.nc
20150922: Confirmed problem reported by Isabelle Dast, reported to
Unidata
20150924: Unidata confirmed problem
20160212: Verified problem still exists in netCDF library
20160512: Ditto
20161028: Verified problem still exists with netCDF 4.4.1
Bug tracking: https://www.unidata.ucar.edu/jira/browse/fxm
More details: http://nco.sf.net/nco.html#ncrename_crd
C. NOT YET FIXED (netCDF4 library bug) in main netCDF release
FIXED in netCDF Development branch as of 20161116
nc-config/nf-config produce erroneous switches that cause NCO builds
to fail
This problem affects netCDF 4.4.1 on all operating systems.
Some pre-compiled netCDF packages may have patched the problem.
Hence it does affect my MacPorts install of netCDF 4.4.1.
Demonstration:
% nc-config --cflags # Produces extraneous text that confuses make
Using nf-config: /usr/local/bin/nf-config
-I/usr/local/include -I/usr/local/include -I/usr/include/hdf
If your nc-config output contains the "Using ..." line, you are
affected by this issue.
20161029: Reported problem to Unidata
20161101: Unidata confirmed reproducibility, attributed to netCDF
4.4.1 changes
20161116: Unidata patch is in tree for netCDF 4.4.2 release
D. NOT YET FIXED (would require DAP protocol change?)
Unable to retrieve contents of variables including period '.' in name
Periods are legal characters in netCDF variable names.
Metadata are returned successfully, data are not.
DAP non-transparency: Works locally, fails through DAP server.
Demonstration:
ncks -O -C -D 3 -v var_nm.dot -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to
find variable
20130724: Verified problem still exists.
Stopped testing because inclusion of var_nm.dot broke all test scripts.
NB: Hard to fix since DAP interprets '.' as structure delimiter in
HTTP query string.
Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47
E. NOT YET FIXED (would require DAP protocol change)
Correctly read scalar characters over DAP.
DAP non-transparency: Works locally, fails through DAP server.
Problem, IMHO, is with DAP definition/protocol
Demonstration:
ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc
20120801: Verified problem still exists
Bug report not filed
Cause: DAP translates scalar characters into 64-element (this
dimension is user-configurable, but still...), NUL-terminated
strings so MD5 agreement fails
"Sticky" reminders:
A. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g.,
HDF4: AMSR MERRA MODIS ...
HDF5: GLAS ICESat Mabel SBUV ...
HDF-EOS5: AURA HIRDLS OMI ...
B. Pre-built executables for many OS's at:
http://nco.sf.net#bnr
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(
