The netCDF Operators NCO version 4.5.1 are ready.
http://nco.sf.net (Homepage)
http://dust.ess.uci.edu/nco (Homepage "mirror")
This release improves regridding features, ncra weighting, and
ncatted flexibility, and contains a raft of minor fixes and tunings.
Notably, most implementation-specific dimension/variable names
involved in regridding can be specified at the command line, and
the regridder understands mapfiles generated by TempestRemap.
ncra can now weight records by a 1-D record variable in the file.
And ncatted supports regular expressions in both the variable name
AND the attribute name (simultaneously, too).
Work on NCO 4.5.2 has commenced and will better support regional
regridding, masks, threading, CF-checking and easier builds on
computer systems that use modules.
Enjoy,
Charlie
NEW FEATURES (full details always in ChangeLog):
A. Regrid global datasets with TempestRemap mapfiles (in addition to
ESMF and SCRIP, which were already supported). Tempest mapfiles
use slightly different conventions than the other two, and this
release accounts for those differences. This capability makes
NCO's regridder a drop-in replacement for the Tempest program
ApplyOfflineMap with global (source and destination) maps.
The two produce the same results, modulo metadata.
Please give it a try and send us feedback!
# Regrid entire file, same output format as input:
ncks --map=map.nc in.nc out.nc
# Deflated netCDF4 output, threading, selected variables:
ncks -4 -L 1 -t 8 -v FS.?,T --map=map.nc in.nc out.nc
http://nco.sf.net/nco.html#regrid
B. ncatted now supports regular expressions in both the variable name
AND the attribute name (simultaneously, too). Previously, ncatted
accepted regular expressions only in the variable name. The new
functionality simplifies sculpting metadata topiary from files with
baroque metadata annotations, since whole groups of attributes may
now be added/modified/deleted with a single command. For example,
delete all attributes whose names end in "_iso19115" from all
variables whose names contain "H2O".
ncatted -a '.?_iso19115$','^H2O*',d,, in.nc
http://nco.sf.net/nco.html#ncatted
C. Add mibs/mabs/mebs methods and operators to ncap2. These are the
absolute-value analogues of min/max/mean, so that, e.g., mibs()
returns the minimum absolute-value of the operand.
tpt_max=temperature.max();
tpt_max=max(temperature);
tpt_mabs=temperature.mabs();
tpt_mabs=mabs(temperature);
http://nco.sf.net/nco.html#mibs
http://nco.sf.net/nco.html#ppc
http://nco.sf.net/nco.html#ncap2
D. Prevent operators from repeating existing cell_methods attribute.
Previously NCO would append new operations, e.g., "time: mean" to
the existing cell_methods attribute, if any. Now it first checks
to see whether that method has been applied and, if so, declines
to write a duplicate operation.
http://nco.sf.net/nco.html#cll_mth
BUG FIXES:
A. Fix formatting of history_of_appended_files attribute when appended
file had no history attribute.
http://nco.sf.net/nco.html#history
B. Round-up to one rather than round-down to zero in chunking maps lfp
and rew. This fixes a bug that only appears on datasets with
certain shapes.
http://nco.sf.net/nco.html#cnk
C. Fix bug using min/max on coordinate variables in ncap2
D. Initialize memory that appeared to cause zeros to be written
by regridder compiled by clang on MacOSX.
KNOWN PROBLEMS DUE TO NCO:
This section of ANNOUNCE reports and reminds users of the
existence and severity of known, not yet fixed, problems.
These problems occur with NCO 4.5.1 built/tested with netCDF
4.4.0-development (20150710) on top of HDF5 hdf5-1.8.13 with:
cd ~/nco;./configure # Configure mechanism -or-
cd ~/nco/bld;make dir;make allinone # Old Makefile mechanism
A. NOT YET FIXED (NCO problem)
Correctly read arrays of NC_STRING with embedded delimiters in
ncatted arguments
Demonstration:
ncatted -D 5 -O -a
new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc
~/foo.nc
ncks -m -C -v att_var ~/foo.nc
20130724: Verified problem still exists
TODO nco1102
Cause: NCO parsing of ncatted arguments is not sophisticated
enough to handle arrays of NC_STRINGS with embedded delimiters.
B. NOT YET FIXED (NCO problem?)
ncra/ncrcat (not ncks) hyperslabbing can fail on variables with
multiple record dimensions
Demonstration:
ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc
20140826: Verified problem still exists
20140619: Problem reported by rmla
Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
Workaround: Convert to fixed dimensions then hyperslab
KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:
A. NOT YET FIXED (netCDF4 or HDF5 problem?)
Specifying strided hyperslab on large netCDF4 datasets leads
to slowdown or failure with recent netCDF versions.
Demonstration with NCO <= 4.4.5:
time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
Demonstration with NCL:
time ncl < ~/nco/data/ncl.ncl
20140718: Problem reported by Parker Norton
20140826: Verified problem still exists
20140930: Finish NCO workaround for problem
Cause: Slow algorithm in nc_var_gets()?
Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
Workaround #2: Convert file to netCDF3 first, then use stride
B. NOT YET FIXED (would require DAP protocol change?)
Unable to retrieve contents of variables including period '.' in name
Periods are legal characters in netCDF variable names.
Metadata are returned successfully, data are not.
DAP non-transparency: Works locally, fails through DAP server.
Demonstration:
ncks -O -C -D 3 -v var_nm.dot -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to
find variable
20130724: Verified problem still exists.
Stopped testing because inclusion of var_nm.dot broke all test scripts.
NB: Hard to fix since DAP interprets '.' as structure delimiter in
HTTP query string.
Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47
C. NOT YET FIXED (would require DAP protocol change)
Correctly read scalar characters over DAP.
DAP non-transparency: Works locally, fails through DAP server.
Problem, IMHO, is with DAP definition/protocol
Demonstration:
ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc
20120801: Verified problem still exists
Bug report not filed
Cause: DAP translates scalar characters into 64-element (this
dimension is user-configurable, but still...), NUL-terminated
strings so MD5 agreement fails
"Sticky" reminders:
A. Pre-built Debian Sid & Ubuntu packages:
http://nco.sf.net#debian
B. Pre-built Fedora and CentOS RPMs:
http://nco.sf.net#rpm
C. Pre-built Mac binaries:
http://nco.sf.net#mac
D. Pre-built Windows (native) and Cygwin binaries:
http://nco.sf.net#windows
E. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g.,
HDF4: AMSR MERRA MODIS ...
HDF5: GLAS ICESat Mabel SBUV ...
HDF-EOS5: AURA HIRDLS OMI ...
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(
