The netCDF Operators NCO version 4.6.6 are ready.
http://nco.sf.net (Homepage, Mailing lists)
http://github.com/nco (Source Code, Releases, Developers)
What's new?
4.6.6 contains a potpurri of new features:
Output 2D rectangular grids formatted as UGRID meshes.
Calendar dates print for bounds and climatology variables.
Revamped HPSS/hsi support. The splitter produces cleaner output.
The regridder groks more files and masks, and annotates output with
more helpful metadata. CDL mode handles compound record variables.
JSON output has changed slightly to re-use NcML keywords.
ncclimo and ncremap have new extract/exclude options.
EVERYONE WHO USES NCKS TO PRINT DATA PLEASE READ THIS:
We would like to change the default behavior of ncks printing.
For 20+ years ncks has, by default, dumped the text representation
of a file in (what we call) "traditional" mode. ncdump produces a
format called CDL that is more useful than traditional mode for most
NCO users. ncks has produced CDL since only 2013 and so CDL is not the
default. ncdump has always printed CDL very cleanly and clearly there
was little point in making CDL the default for ncks.
Now ncks CDL mode rivals ncdump, and is superior in some ways.
In particular, ncks now prints times as human-readable calendar dates,
the last ncdump feature I used that ncks did not have.
We propose to change the default printed output of ncks to CDL.
Then one would type simply "ncks" instead of "ncks --cdl".
A new "--trd" option would still print traditional output.
Let us know if this bothers or pleases you prior to 4.6.7.
You have been warned :)
Work on NCO 4.6.7 has commenced. Planned improvements include
support for conda installs on MS Windows, more ncclimo and ncremap
features.
Enjoy,
Charlie
NEW FEATURES (full details always in ChangeLog):
A. ncremap and ncclimo support new extract/exclude switches:
--no_cll_msr: Exclude cell_measures variables
--no_frm_trm: Exclude formula_terms variables
--no_ntv_tms: Exclude native-grid timeseries (splitter only)
--no_stg_grd: Exclude staggered grid coordinates
By default, these variables are always extracted.
Use the above switches to exclude them instead, e.g.,
ncclimo --no_cll_msr --no_frm_trm --no_ntv_tms --no_stg_grd ...
http://nco.sf.net/nco.html#no_cll_msr
http://nco.sf.net/nco.html#no_frm_trm
http://nco.sf.net/nco.html#no_ntv_tms
http://nco.sf.net/nco.html#no_stg_grd
B. ncclimo's splitter mode no longer requires variable lists.
Previously the splitter required a '-v var_lst' option to know
which variables to split into timeseries. Now when the -v option
is omitted the splitter creates timeseries of all variables of
rank R >= 2. Typically this includes all spatial variables.
Variables known to be extensive (like grid properties) are
automatically excluded from this list. However, we expect there
will be some cases where the splitter creates timeseries from
R >= 2 variables that are not desirable. We are happy to encode
manual exceptions as we receive notice of them.
ncclimo -s 1979 -e 1981 -o $drc_out in*.nc
http://nco.sf.net/nco.html#ncclimo
C. The regridder now produces UGRID output for 2D rectangular grids.
Invoke ncremap in grid inferral mode with "--rgr ugrid=foo.nc"
and voilà, an annoted UGRID file is produce along with the
default SCRIP gridfile. The NCO UGRID is more complete than that
produced by ESMF_Scrip2Unstruct, but ESMF can create UGRID for much
more comlex grids. The NCO regridder can currently only generate
UGRID for rectangular 2D input grids, and there are no plans to
extend this to curvilinear or unstructured input grids.
ncks --rgr nfr=Y --rgr grid=scrip.nc --rgr ugrid=ugrid.nc in.nc out.nc
http://nco.sf.net/nco.html#rgr
D. ncks now prints the units, if any, in comments after CDL values
when debugging is enabled with dbg_lvl >=1:
zender@foo: ncks --cdl -D 1 in.nc
lat = -90, 90 ; // units="degrees_north"
lon = 0, 90, 180, 270 ; // units="degrees_east"
tpt = 273.1, 273.2, ... 274 ; // units="kelvin"
http://nco.sf.net/nco.html#cdl
E. ncap2 now implements the CF cell_methods convention.
ncra, nces, and ncwa have long implemented this, and now the
relevant ncap2 methods (avg, min, max, etc.) record the operation
type in the cell_methods attribute.
ncap2 -s 'avg=one_dmn_rec_var.avg()' ~/nco/data/in.nc ~/foo.nc
http://nco.sf.net/nco.html#cll_mth
F. Since 4.6.5 ncks has printed the human-legible calendar string
corresponding to values with UDUnits date units (time since
basetime, e.g., "days since 2000-01-01") and CF calendar attribute,
if any. Now ncks also translates and prints time values associated
with CF climatology and bounds variables as human legible dates.
As before, enact this with the --calendar option when printing.
Using dbg_lvl >= 1 in CDL mode prints both the value and calendar
string (one in comments):
zender@aerosol:~$ ncks -D 1 --cdl --cal -v tm_365 ~/nco/data/in.nc
tm_365 = "2013-03-01"; // double value: 59
zender@aerosol:~$ ncks -D 1 --cdl -v tm_365 ~/nco/data/in.nc
tm_365 = 59; // calendar format: "2013-03-01"
This option is similar to the ncdump -t option.
Does this work how you'd like? Let us know.
http://nco.sf.net/nco.html#cal
G. ncks now prints "(no units)" instead of "(null)" as the units
string for variables with no units. And in CDL mode ncks, when
invoked with debugging (e.g., -D 1) prints the units string in the
comment section after the values of a variable.
BUG FIXES:
A. NCO now exits with a clear message instead of core dump on missing
files on HPSS systems with the hsi command.
B. ncremap will no longer define mask variables in inferred grid
files unless it finds a mask variable in the source file.
Previously ncremap could define a mask variable in the inferred
grid file and never fill-it with data.
C. ncremap now invokes the neareststod algorithm when requested.
Previously a typo caused ncremap to invoke nearestdtos when
neareststod was specified (doh!). Solution is to upgrade.
D. ncremap now converts non-default masks correctly in infer mode.
Previously a "mask" variable in an input file whose grid structure
was being inferred could be incorrectly handled for some numeric
types. This likely only affected masks with missing values.
Sources of files possibly affected include AMSR and CICE.
E. ncks now prints braces required to disambiguate compound
record dimensions in CDL mode. Such braces have been required by
ncgen-like parsers since ~2013 in order to unambiguously determine
the record dimension sizes in variables with multiple record
dimensions. This fix only affects CDL printing of netCDF4 files
that contain compound record variables.
KNOWN PROBLEMS DUE TO NCO:
This section of ANNOUNCE reports and reminds users of the
existence and severity of known, not yet fixed, problems.
These problems occur with NCO 4.6.6 built/tested under
MacOS 10.12.3 with netCDF 4.4.1 on HDF5 1.8.16 and with
Linux with netCDF 4.4.2-development (20161116) on HDF5 1.8.16.
A. NOT YET FIXED (NCO problem)
Correctly read arrays of NC_STRING with embedded delimiters in
ncatted arguments
Demonstration:
ncatted -D 5 -O -a
new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc
~/foo.nc
ncks -m -C -v att_var ~/foo.nc
20130724: Verified problem still exists
TODO nco1102
Cause: NCO parsing of ncatted arguments is not sophisticated
enough to handle arrays of NC_STRINGS with embedded delimiters.
B. NOT YET FIXED (NCO problem?)
ncra/ncrcat (not ncks) hyperslabbing can fail on variables with
multiple record dimensions
Demonstration:
ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc
20140826: Verified problem still exists
20140619: Problem reported by rmla
Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
Workaround: Convert to fixed dimensions then hyperslab
KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:
A. NOT YET FIXED (netCDF4 or HDF5 problem?)
Specifying strided hyperslab on large netCDF4 datasets leads
to slowdown or failure with recent netCDF versions.
Demonstration with NCO <= 4.4.5:
time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
Demonstration with NCL:
time ncl < ~/nco/data/ncl.ncl
20140718: Problem reported by Parker Norton
20140826: Verified problem still exists
20140930: Finish NCO workaround for problem
Cause: Slow algorithm in nc_var_gets()?
Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
Workaround #2: Convert file to netCDF3 first, then use stride
B. NOT YET FIXED (netCDF4 library bug)
Simultaneously renaming multiple dimensions in netCDF4 file can
corrupt output
Demonstration:
ncrename -O -d lev,z -d lat,y -d lon,x ~/nco/data/in_grp.nc ~/foo.nc
# Completes but file is unreadable
ncks -v one ~/foo.nc
20150922: Confirmed problem reported by Isabelle Dast, reported to
Unidata
20150924: Unidata confirmed problem
20160212: Verified problem still exists in netCDF library
20160512: Ditto
20161028: Verified problem still exists with netCDF 4.4.1
20170323: Verified problem still exists with netCDF 4.4.2-development
Bug tracking: https://www.unidata.ucar.edu/jira/browse/fxm
More details: http://nco.sf.net/nco.html#ncrename_crd
C. NOT YET FIXED (netCDF4 library bug)
Renaming a non-coordinate variable to a coordinate variable fails in
netCDF4
Demonstration:
ncrename -O -v non_coord,coord ~/nco/data/in_grp.nc ~/foo.nc # Fails
(HDF error)
20170323: Confirmed problem reported by Paolo Oliveri, reported to
Unidata
Bug tracking: https://www.unidata.ucar.edu/jira/browse/fxm
More details: http://nco.sf.net/nco.html#ncrename_crd
D. FIXED in netCDF Development branch as of 20161116 and in maintenance
release 4.4.1.1
nc-config/nf-config produce erroneous switches that cause NCO builds
to fail
This problem affects netCDF 4.4.1 on all operating systems.
Some pre-compiled netCDF packages may have patched the problem.
Hence it does not affect my MacPorts install of netCDF 4.4.1.
Demonstration:
% nc-config --cflags # Produces extraneous text that confuses make
Using nf-config: /usr/local/bin/nf-config
-I/usr/local/include -I/usr/local/include -I/usr/include/hdf
If your nc-config output contains the "Using ..." line, you are
affected by this issue.
20161029: Reported problem to Unidata
20161101: Unidata confirmed reproducibility, attributed to netCDF
4.4.1 changes
20161116: Unidata patch is in tree for netCDF 4.4.2 release
20161123: Fixed in maintenance release netCDF 4.4.1.1
E. NOT YET FIXED (would require DAP protocol change?)
Unable to retrieve contents of variables including period '.' in name
Periods are legal characters in netCDF variable names.
Metadata are returned successfully, data are not.
DAP non-transparency: Works locally, fails through DAP server.
Demonstration:
ncks -O -C -D 3 -v var_nm.dot -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to
find variable
20130724: Verified problem still exists.
Stopped testing because inclusion of var_nm.dot broke all test scripts.
NB: Hard to fix since DAP interprets '.' as structure delimiter in
HTTP query string.
Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47
F. NOT YET FIXED (would require DAP protocol change)
Correctly read scalar characters over DAP.
DAP non-transparency: Works locally, fails through DAP server.
Problem, IMHO, is with DAP definition/protocol
Demonstration:
ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc
20120801: Verified problem still exists
Bug report not filed
Cause: DAP translates scalar characters into 64-element (this
dimension is user-configurable, but still...), NUL-terminated
strings so MD5 agreement fails
"Sticky" reminders:
A. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g.,
HDF4: AMSR MERRA MODIS ...
HDF5: GLAS ICESat Mabel SBUV ...
HDF-EOS5: AURA HIRDLS OMI ...
B. Pre-built executables for many OS's at:
http://nco.sf.net#bnr
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(
