The netCDF Operators NCO version 4.6.5 are ready.
http://nco.sf.net (Homepage, Mailing lists)
http://github.com/nco (Source Code, Releases, Developers)
What's new?
4.6.5 contains a potpurri of new features. Setting chunk cache size
with --cnk_csh can accelerate aggregation of netCDF4 files.
Climatology and regridder users may appreciate the cleaned-up
output offered by the ncremap --vrb_lvl switch and more intelligent
grid inferral. Everyone who has tried to convert "time since..."
values to calendar dates will like the new ncks --cal switch that does
the UDUnits conversion for you. There are also helpful bugfixes for
ncap2 attribute propagation in corner cases, and corrected error codes
for bad command line arguments.
Work on NCO 4.6.6 has commenced. Planned improvements include
support for conda installs on MS Windows, and human-legible calendar
date printing, more ncclimo and ncremap features.
Enjoy,
Charlie
NEW FEATURES (full details always in ChangeLog):
A. ncks allows setting chunk cache-size with --cnk_csh with all
applicable operators. This sets the cache size for all variables.
Users have demonstrated that increasing cache size can dramatically
speed-up chunking operations on large datasets.
This setting only works on netCDF4 files.
ncrcat --cnk_csh=1000000000 --cnk_plc=g3d --cnk_dmn=time,365 \
--cnk_dmn=lat,1800 --cnk_dmn=lon,3600 in*.nc4 out.nc4
http://nco.sf.net/nco.html#cnk_csh
Thanks to Hugo Oliveira for encouraging this feature
B. ncks now prints the human-legible calendar string corresponding to
values with UDUnits date units (time since basetime, e.g., "days
since 2000-01-01") and CF calendar attribute, if any. Enact this
with the --calendar option when printing. Using dbg_lvl >= 1 in CDL
mode prints both the value and calendar string (one in comments):
zender@aerosol:~$ ncks -D 1 --cdl --cal -v tm_365 ~/nco/data/in.nc
tm_365 = "2013-03-01"; // double value: 59
zender@aerosol:~$ ncks -D 1 --cdl -v tm_365 ~/nco/data/in.nc
tm_365 = 59; // calendar format: "2013-03-01"
This option is similar to the ncdump -t option.
Does this work how you'd like? Let us know.
http://nco.sf.net/nco.html#cal
C. ncap2 now seamlessly edits character arrays.
Suppose you have a string stored as an old school character array
(not a netCDF4 string) with a fixed dimension size. ncap2 now
supports altering the string value without having to manually pad
the RHS to the exact dimension size of the LHS:
ncap2 -s 'date_sng="2016-05-11_06:00:00"' in.nc out.nc
ncap2 -s 'date_sng="automatic padding"' in.nc out.nc
The exact same syntax has always worked on netCDF4 strings.
This new feature frees users from needing to know the ugly details
of netCDF3 string storage.
http://nco.sf.net/nco.html#ncap2
D. ncremap support a verbosity level option, --vrb_lvl.
Verbosity level is now independent of debug level, dbg_lvl, so
users can customize how much ncremap information to print.
Default vrb_lvl=2 and supported levels range from 0 (no output) to
4 (most verbose). The verbosity of the underlying remapping,
which invokes ncks, is still controlled by dbg_lvl not vrb_lvl.
ncremap --vrb_lvl=1 -i in.nc -m map.nc -o out.nc
http://nco.sf.net/nco.html#ncremap
Thanks to Milena Veneziani for requesting.
E. ncremap and ncclimo now support a version option, --version.
This option prints the operator version and configuration.
It's a painless way to tell, e.g., if ncremap knows where your
ESMF_RegridWeightGen executable is, and where these scripts
look for NCO and what version of netCDF is linked to:
ncremap --version
ncclimo --version
http://nco.sf.net/nco.html#ncremap
http://nco.sf.net/nco.html#ncclimo
F. ncremap and ncks now parse more unknown grids than before.
The front-end of ncremap's grid-inferenence routine has been
refactored to rely on less user-provided information about
dimension names. Now it obtains more information than before
by examining the grid coordinates. As a result, grids from
data files with strange dimension names are likelier to be parsed.
BUG FIXES:
A. ncap2 attribute propagation for some unary operators is fixed.
It had been broken since ~4.6.3 for unary operations like 'time+=10'
whereas it was working fine for binary operations, e.g., 'T=T+10'.
Thanks to Qi Tang for sounding the alert. Solution is to upgrade.
B. The chunk cache option (--cnk_csh) implemented in ncks in 4.6.4 was
inadvertently doing a no-op. Now it works as advertised.
No workaround, solution is to upgrade.
C. A broken diagnostic introduced in 4.6.3 or 4.6.4 could cause the
regridder to fail. This occurred with some POP files. Diagnostic
now turned-off. Solution is to upgrade, workaround is use NCO <=
4.6.2. Thanks to Sungduk Yu for reporting.
D. Bad command-line options now always return with EXIT_FAILURE (1).
Previously they could return with EXIT_SUCCESS (0).
Thanks to Jerome Mao for reporting.
KNOWN PROBLEMS DUE TO NCO:
This section of ANNOUNCE reports and reminds users of the
existence and severity of known, not yet fixed, problems.
These problems occur with NCO 4.6.5 built/tested under
MacOS 10.12.1 with netCDF 4.4.1 on HDF5 1.8.16 and with
Linux with netCDF 4.4.2-development (20161116) on HDF5 1.8.16.
A. NOT YET FIXED (NCO problem)
Correctly read arrays of NC_STRING with embedded delimiters in
ncatted arguments
Demonstration:
ncatted -D 5 -O -a
new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc
~/foo.nc
ncks -m -C -v att_var ~/foo.nc
20130724: Verified problem still exists
TODO nco1102
Cause: NCO parsing of ncatted arguments is not sophisticated
enough to handle arrays of NC_STRINGS with embedded delimiters.
B. NOT YET FIXED (NCO problem?)
ncra/ncrcat (not ncks) hyperslabbing can fail on variables with
multiple record dimensions
Demonstration:
ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc
20140826: Verified problem still exists
20140619: Problem reported by rmla
Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
Workaround: Convert to fixed dimensions then hyperslab
KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:
A. NOT YET FIXED (netCDF4 or HDF5 problem?)
Specifying strided hyperslab on large netCDF4 datasets leads
to slowdown or failure with recent netCDF versions.
Demonstration with NCO <= 4.4.5:
time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
Demonstration with NCL:
time ncl < ~/nco/data/ncl.ncl
20140718: Problem reported by Parker Norton
20140826: Verified problem still exists
20140930: Finish NCO workaround for problem
Cause: Slow algorithm in nc_var_gets()?
Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
Workaround #2: Convert file to netCDF3 first, then use stride
B. NOT YET FIXED (netCDF4 library bug)
Simultaneously renaming multiple dimensions in netCDF4 file can
corrupt output
Demonstration:
ncrename -O -d lev,z -d lat,y -d lon,x ~/nco/data/in_grp.nc ~/foo.nc
# Completes but file is unreadable
ncks -v one ~/foo.nc
20150922: Confirmed problem reported by Isabelle Dast, reported to
Unidata
20150924: Unidata confirmed problem
20160212: Verified problem still exists in netCDF library
20160512: Ditto
20161028: Verified problem still exists with netCDF 4.4.1
Bug tracking: https://www.unidata.ucar.edu/jira/browse/fxm
More details: http://nco.sf.net/nco.html#ncrename_crd
C. FIXED in netCDF Development branch as of 20161116 and in maintenance
release 4.4.1.1
nc-config/nf-config produce erroneous switches that cause NCO builds
to fail
This problem affects netCDF 4.4.1 on all operating systems.
Some pre-compiled netCDF packages may have patched the problem.
Hence it does not affect my MacPorts install of netCDF 4.4.1.
Demonstration:
% nc-config --cflags # Produces extraneous text that confuses make
Using nf-config: /usr/local/bin/nf-config
-I/usr/local/include -I/usr/local/include -I/usr/include/hdf
If your nc-config output contains the "Using ..." line, you are
affected by this issue.
20161029: Reported problem to Unidata
20161101: Unidata confirmed reproducibility, attributed to netCDF
4.4.1 changes
20161116: Unidata patch is in tree for netCDF 4.4.2 release
20161123: Fixed in maintenance release netCDF 4.4.1.1
D. NOT YET FIXED (would require DAP protocol change?)
Unable to retrieve contents of variables including period '.' in name
Periods are legal characters in netCDF variable names.
Metadata are returned successfully, data are not.
DAP non-transparency: Works locally, fails through DAP server.
Demonstration:
ncks -O -C -D 3 -v var_nm.dot -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to
find variable
20130724: Verified problem still exists.
Stopped testing because inclusion of var_nm.dot broke all test scripts.
NB: Hard to fix since DAP interprets '.' as structure delimiter in
HTTP query string.
Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47
E. NOT YET FIXED (would require DAP protocol change)
Correctly read scalar characters over DAP.
DAP non-transparency: Works locally, fails through DAP server.
Problem, IMHO, is with DAP definition/protocol
Demonstration:
ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc
20120801: Verified problem still exists
Bug report not filed
Cause: DAP translates scalar characters into 64-element (this
dimension is user-configurable, but still...), NUL-terminated
strings so MD5 agreement fails
"Sticky" reminders:
A. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g.,
HDF4: AMSR MERRA MODIS ...
HDF5: GLAS ICESat Mabel SBUV ...
HDF-EOS5: AURA HIRDLS OMI ...
B. Pre-built executables for many OS's at:
http://nco.sf.net#bnr
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(
