The netCDF Operators NCO version 4.5.2 are ready.
http://nco.sf.net (Homepage, Mailing lists)
http://github.com/nco (Source Code, Releases, Developers)
First, NCO now resides at http://github.com/nco, i.e., it is a
GitHub organization. We are pleased to co-habit with the pyNCO
project of Joe Hamman who has produced NCO Python bindings.
Please check 'em out. All GitHub paths with "czender" instead of
"nco" as the username will still work, but should be updated.
Change your local copy of the repository with:
git remote set-url origin https://github.com/nco/nco.git
The NCO homepage and discussion forums continue to reside on
SourceForge. That is still the preferred place to ask questions
and post bug-reports. NCO source code, though, is only on GitHub.
This NCO release improves regridding features, ncra weighting, and
ncatted flexibility, and contains a raft of minor fixes and tunings.
Also, in case you need updated NCO versions at US computing centers,
know that we now maintain up-to-date NCO in ~zender/[bin,lib] at
ANL/ALCF (cooley), NCAR/NWSC (yellowstone), LBNL/NERSC (edison), and
ORNL/OLCF (rhea).
Work on NCO 4.5.3 has commenced and will better support regional
regridding, masks, threading, CF-checking and easier builds on
computer systems that use modules.
Enjoy,
Charlie
NEW FEATURES (full details always in ChangeLog):
A. All operators can now add user-specified global attributes to
output files using the --glb switch. The switch takes mandatory
arguments --glb att_nm=att_val where att_nm is the desired name of
the global attriute to add, and att_val is its value.
Currently only text attributes are supported (recorded as type
NC_CHAR, and regular expressions are not allowed (unlike ncatted).
Multiple invocations simplify the annotation of output file at
creation (or modification) time, and incur no performance penalty.
Should users emit a loud hue and cry, we will consider ading the
the functionality of ncatted to the front-end of all operators,
i.e., accepting valid arguments to modify attributes of any type
and to apply regular expressions.
ncra --glb machine=${HOSTNAME} --glb created_by=${USER} in*.nc out.nc
http://nco.sf.net/nco.html#glb
B. Grid generation: ncks generates SCRIP-format gridfiles for
select grid types, in cluding uniform/equi-angular, cap/FV, and
Gaussian. Options pertinent to the grid geometry and metadata are
passed to NCO via key-value pairs prefixed by "--rgr". Pass at
least six key-value pair arguments to create a fully explicitly
specified, well-annotated grid. The six arguments, and their
corresponding keys in parentheses, are: grid title (grd_ttl),
filename to write the grid to (grid), number of latitudes
(lat_nbr), number of longitudes (lon_nbr), latitude grid type
(lat_typ), and longitude grid type (lon_typ). Four other arguments
(the NSEW bounding box) are necessary to construct regular regional
(non-global) grids.
ncks -grid=FV129x256.nc \
--rgr lat_nbr=129 --rgr lon_nbr=256 --rgr lat_typ=cap
--rgr lon_typ=grn_ctr ~zender/nco/data/in.nc ~/foo.nc
ncks -grid=T42.nc \
--rgr lat_nbr=64 --rgr lon_nbr=128 --rgr lat_typ=gss
--rgr lon_typ=grn_ctr ~zender/nco/data/in.nc ~/foo.nc
http://nco.sf.net/nco.html#grid
C. Regrid regional datasets with ESMF, SCRIP, and TempestRemap
mapfiles. Please give it a try and send us feedback!
# Regrid global file to regional domain
ncks --map map_conusx4v1_to_fv0.1x0.1_US_bilin.nc in.nc out.nc
http://nco.sf.net/nco.html#regrid
D. Multi-file operators support a new extension to the -n option
that understands calendar month counting. It is now possibly to
automatically generate series of filenames whose right-most two
digits increment from a specified minimum (e.g., 03) by a specified
amount (e.g., 1) until a specified maximum (e.g., 05) is reached
and then the leftmost digits (i.e., the year) increments by one,
and the whole process is reapeated until the total specified number
of filenames is generated.
ncrcat -n 3,6,1,12,1,yyyymm 198512.nc 198512_198602.nc
http://nco.sf.net/nco.html#input
BUG FIXES:
A. The ncra --wgt option, introduced in 4.4.9, works fine except,
it turns out, for variables with missing values. The normalization
for these values was incorrect and produced errors whose size was
roughly inversely proportional to the size of the weights.
The solution is to upgrade to 4.5.2. Thanks to Jeff Painter for
reporting this problem.
KNOWN PROBLEMS DUE TO NCO:
This section of ANNOUNCE reports and reminds users of the
existence and severity of known, not yet fixed, problems.
These problems occur with NCO 4.5.2-alpha7 built/tested with netCDF
4.4.0-development (20150818) on top of HDF5 hdf5-1.8.13 with:
cd ~/nco;./configure # Configure mechanism -or-
cd ~/nco/bld;make dir;make allinone # Old Makefile mechanism
A. NOT YET FIXED (NCO problem)
Correctly read arrays of NC_STRING with embedded delimiters in
ncatted arguments
Demonstration:
ncatted -D 5 -O -a
new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc
~/foo.nc
ncks -m -C -v att_var ~/foo.nc
20130724: Verified problem still exists
TODO nco1102
Cause: NCO parsing of ncatted arguments is not sophisticated
enough to handle arrays of NC_STRINGS with embedded delimiters.
B. NOT YET FIXED (NCO problem?)
ncra/ncrcat (not ncks) hyperslabbing can fail on variables with
multiple record dimensions
Demonstration:
ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc
20140826: Verified problem still exists
20140619: Problem reported by rmla
Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
Workaround: Convert to fixed dimensions then hyperslab
KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:
A. NOT YET FIXED (netCDF4 or HDF5 problem?)
Specifying strided hyperslab on large netCDF4 datasets leads
to slowdown or failure with recent netCDF versions.
Demonstration with NCO <= 4.4.5:
time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
Demonstration with NCL:
time ncl < ~/nco/data/ncl.ncl
20140718: Problem reported by Parker Norton
20140826: Verified problem still exists
20140930: Finish NCO workaround for problem
Cause: Slow algorithm in nc_var_gets()?
Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
Workaround #2: Convert file to netCDF3 first, then use stride
B. NOT YET FIXED (would require DAP protocol change?)
Unable to retrieve contents of variables including period '.' in name
Periods are legal characters in netCDF variable names.
Metadata are returned successfully, data are not.
DAP non-transparency: Works locally, fails through DAP server.
Demonstration:
ncks -O -C -D 3 -v var_nm.dot -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to
find variable
20130724: Verified problem still exists.
Stopped testing because inclusion of var_nm.dot broke all test scripts.
NB: Hard to fix since DAP interprets '.' as structure delimiter in
HTTP query string.
Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47
C. NOT YET FIXED (would require DAP protocol change)
Correctly read scalar characters over DAP.
DAP non-transparency: Works locally, fails through DAP server.
Problem, IMHO, is with DAP definition/protocol
Demonstration:
ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc
20120801: Verified problem still exists
Bug report not filed
Cause: DAP translates scalar characters into 64-element (this
dimension is user-configurable, but still...), NUL-terminated
strings so MD5 agreement fails
"Sticky" reminders:
A. Pre-built Debian Sid & Ubuntu packages:
http://nco.sf.net#debian
B. Pre-built Fedora and CentOS RPMs:
http://nco.sf.net#rpm
C. Pre-built Mac binaries:
http://nco.sf.net#mac
D. Pre-built Windows (native) and Cygwin binaries:
http://nco.sf.net#windows
E. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g.,
HDF4: AMSR MERRA MODIS ...
HDF5: GLAS ICESat Mabel SBUV ...
HDF-EOS5: AURA HIRDLS OMI ...
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(
