The netCDF Operators NCO version 4.6.7 are ready.
http://nco.sf.net (Homepage, Mailing lists)
http://github.com/nco (Source Code, Releases, Developers)
What's new?
The main new features in 4.6.7 are in the regridder, which
introduces a sub-grid regridding option, more automated handling
and flexibility with regional grids, and easier flag specification.
ncap2 added specialty functions to simplify handling of missing
values and in-filling large missing regions with interpolated data.
ncatted received an important bugfix for netCDF4 with _FillValue.
PLEASE READ THIS IF YOU USE NCKS TO PRINT DATA:
The ncks default printing will change to CDL in summer 2017.
For 20+ years ncks has, by default, dumped the text representation
of a file in (what we call) "traditional" mode, which best facilitates
meticulous data examination in line-by-line format.
ncdump produces CDL format that is more useful for most NCO users.
ncdump has always printed clean CDL and for many years there was
little point in defaulting ncks printed output to CDL.
However, ncks CDL mode now rivals ncdump in many ways.
In particular, ncks --cln now prints times as human-readable calendar
dates, the last ncdump feature that I used which ncks lacked.
Hence in summer 2017 ncks default printed output will change to CDL.
Then one will type simply "ncks" instead of "ncks --cdl".
We just added a new "--trd" option to print traditional output.
User --trd in your scripts and their behavior will not change.
Otherwise your printing scripts will start to print CDL this summer.
You have been warned :)
Work on NCO 4.6.8 has commenced. Planned improvements include
support for conda installs on MS Windows, more ncclimo and ncremap
features, more ncap2 chunking flexibility.
Enjoy,
Charlie
NEW FEATURES (full details always in ChangeLog):
A. ncremap has a new sub-gridscale (SGS) mode, -P sgs.
In sub-grid mode, ncremap performs substantial pre- and
post-processing so that regridding conserves fields that may
represent only a fraction of the entire gridcell.
The sub-gridscale fraction represented by each field is contained
in variable whose can be set with the new --sgs_frc option
(default is "landfrac"). SGS mode eases regridding of datasets
(e.g., from ALM, CLM, CICE) that output data normalized to a
fraction (rather than the entire extent) of the gridcell.
SGS mode automatically derives new binary masks (--sgs_msk)
and allows for additional normalization (--sgs_nrm).
Specific flavors of SGS can be selected (-P alm, clm, or cice).
These ensure regridded datasets recreate the idiosyncratic units
(e.g., %, km2) employed by raw ALM, CLM, and CICE model output.
Discerning these regridded from raw datasets is difficult :)
ncremap -P sgs -s src_grd.nc -i in.nc -d 1x1.nc -o rgr.nc
ncremap -P alm -s src_grd.nc -i in.nc -d 1x1.nc -o rgr.nc
ncremap -P cice -s src_grd.nc -i in.nc -d 1x1.nc -o rgr.nc
ncremap -P sgs --sgs_frc=aice --sgs_msk=tmask --src_nrm=100 ...
http://nco.sf.net/nco.html#sgs
Thanks to Forrest Hoffmann and Min Xu for prototyping the
algorithm and verifying the results.
B. ncremap now automatically indicates to the ERWG weight generator
when either or both source and destination grids are regional.
This information can considerably reduce the time and space
the weight generator requires to generate and store the map-file.
ncremap uses fuzzy logic to determine this for inferred grids.
The user may inform ncremap whether supplied grids (i.e.,
specified with -s or -g) are regional with the new --rgn_dst
and --rgn_src flags:
ncremap -i in.nc -d dst.nc -m map.nc # Inferred (no option necessary)
ncremap -s grd_src.nc -g grd_dst.nc -m map.nc --rgn_src # Supplied
Users may notice significantly faster throughput due to this.
http://nco.sf.net/nco.html#rgn_src
C. ncremap switches have been "rationalized" to eliminate poor UI
decisions made in early development. Please read the old/new
option meanings and change your scripts accordingly. Now -a
subsumes the old role of -w. HINT: use long-options (e.g.,
--algorithm=conserve) to future-proof your scripts.
Meaning of -a:
Old: Interpolation algorithm for ESMF only
New: Interpolation algorithm for ESMF and Tempest ('tempest')
Meaning of -E:
Old: Options for ESMF_RegridWeightGen (now use -W)
New: Unused
Meaning of -T:
Old: Options for TempestRemap (now use -W)
New: temporary directory (replaces old -U)
Meaning of -U:
Old: temporary directory
New: unpack flag (tells ncremap to unpack data before regridding)
Meaning of -w:
Old: Weight-generator keyword. Valid options were 'esmf' and 'tempest'.
New: Weight-generator command string (e.g., "mpirun
ESMF_RegridWeightGen -n 24")
Supercedes default 'ESMF_RegridWeightGen' or 'GenerateOfflineMap'
Meaning of -W:
Old: Unused
New: Options for weight-generator (replaces old -E and -T)
http://nco.sf.net/nco.html#ncremap
D. ncap2 has a new function named missing(), aka mask_miss() that
returns an array of the same size of its argument, with values
of zero where valid data are present and one where missing data
are present. The resulting array can be useful in characterizing
and masking the original array.
ncap2 -s 'mss_val=three_dmn_var_dbl.missing().ttl($time)' in.nc out.nc
Thanks to Adrian Tompkins for suggesting this feature.
http://nco.sf.net/nco.html#mask_miss
E. ncap2 now propagates existing chunking parameters from the input
file to the output file, and similarly chunks created variables.
The rest of NCO has long implemented comprehensive chunking
features and now ncap2 supports the basic methods.
ncap2 -s 'avg=one_dmn_rec_var.avg()' ~/nco/data/in.nc ~/foo.nc
http://nco.sf.net/nco.html#cll_mth
F. ncks now has a --trd (or --traditional) option to print output
in the "tradiational" format. Start to use this unless you prefer
CDL output which will become the default in summer 2017, possibly
in NCO 4.7.0. At that time, the --cdl option will become default.
BUG FIXES:
A. ncatted allows writing an empty NC_CHAR or NC_STRING attribute.
Previously ncatted would core dump. ncks also prints such values.
No workaround. Solution is to upgrade.
B. ncap2 could die when performing computations with pre-chunked
and/or deflated data. Workaround is to convert to netCDF3
before computations. Solution is to upgrade.
Thanks to Dave Blodgett for reporting this issue.
C. ncatted now correctly handles creation, modification, overwriting
of _FillValue attributes in netCDF4 data. Since about 4.6.2
fiddling with netCDF4 _FillValue attributes with existing data
could lead to data loss. The workaround is to convert to netCDF3,
fiddle, then convert back to netCDF4. The solution is to upgrade.
Thanks to Kyle Wilcox for reporting this issue.
KNOWN PROBLEMS DUE TO NCO:
This section of ANNOUNCE reports and reminds users of the
existence and severity of known, not yet fixed, problems.
These problems occur with NCO 4.6.7 built/tested under
MacOS 10.12.5 with netCDF 4.4.1 on HDF5 1.10.0-patch1 and with
Linux with netCDF 4.4.2-development (20170322) on HDF5 1.8.17.
A. NOT YET FIXED (NCO problem)
Correctly read arrays of NC_STRING with embedded delimiters in
ncatted arguments
Demonstration:
ncatted -D 5 -O -a
new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc
~/foo.nc
ncks -m -C -v att_var ~/foo.nc
20130724: Verified problem still exists
TODO nco1102
Cause: NCO parsing of ncatted arguments is not sophisticated
enough to handle arrays of NC_STRINGS with embedded delimiters.
B. NOT YET FIXED (NCO problem?)
ncra/ncrcat (not ncks) hyperslabbing can fail on variables with
multiple record dimensions
Demonstration:
ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc
20140826: Verified problem still exists
20140619: Problem reported by rmla
Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
Workaround: Convert to fixed dimensions then hyperslab
KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:
A. NOT YET FIXED (netCDF4 or HDF5 problem?)
Specifying strided hyperslab on large netCDF4 datasets leads
to slowdown or failure with recent netCDF versions.
Demonstration with NCO <= 4.4.5:
time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
Demonstration with NCL:
time ncl < ~/nco/data/ncl.ncl
20140718: Problem reported by Parker Norton
20140826: Verified problem still exists
20140930: Finish NCO workaround for problem
Cause: Slow algorithm in nc_var_gets()?
Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
Workaround #2: Convert file to netCDF3 first, then use stride
B. NOT YET FIXED (netCDF4 library bug)
Simultaneously renaming multiple dimensions in netCDF4 file can
corrupt output
Demonstration:
ncrename -O -d lev,z -d lat,y -d lon,x ~/nco/data/in_grp.nc ~/foo.nc
# Completes but file is unreadable
ncks -v one ~/foo.nc
20150922: Confirmed problem reported by Isabelle Dast, reported to
Unidata
20150924: Unidata confirmed problem
20160212: Verified problem still exists in netCDF library
20160512: Ditto
20161028: Verified problem still exists with netCDF 4.4.1
20170323: Verified problem still exists with netCDF 4.4.2-development
Bug tracking: https://www.unidata.ucar.edu/jira/browse/fxm
More details: http://nco.sf.net/nco.html#ncrename_crd
C. NOT YET FIXED (netCDF4 library bug)
Renaming a non-coordinate variable to a coordinate variable fails in
netCDF4
Demonstration:
ncrename -O -v non_coord,coord ~/nco/data/in_grp.nc ~/foo.nc # Fails
(HDF error)
20170323: Confirmed problem reported by Paolo Oliveri, reported to
Unidata
Bug tracking: https://www.unidata.ucar.edu/jira/browse/fxm
More details: http://nco.sf.net/nco.html#ncrename_crd
D. FIXED in netCDF Development branch as of 20161116 and in maintenance
release 4.4.1.1
nc-config/nf-config produce erroneous switches that cause NCO builds
to fail
This problem affects netCDF 4.4.1 on all operating systems.
Some pre-compiled netCDF packages may have patched the problem.
Hence it does not affect my MacPorts install of netCDF 4.4.1.
Demonstration:
% nc-config --cflags # Produces extraneous text that confuses make
Using nf-config: /usr/local/bin/nf-config
-I/usr/local/include -I/usr/local/include -I/usr/include/hdf
If your nc-config output contains the "Using ..." line, you are
affected by this issue.
20161029: Reported problem to Unidata
20161101: Unidata confirmed reproducibility, attributed to netCDF
4.4.1 changes
20161116: Unidata patch is in tree for netCDF 4.4.2 release
20161123: Fixed in maintenance release netCDF 4.4.1.1
E. NOT YET FIXED (would require DAP protocol change?)
Unable to retrieve contents of variables including period '.' in name
Periods are legal characters in netCDF variable names.
Metadata are returned successfully, data are not.
DAP non-transparency: Works locally, fails through DAP server.
Demonstration:
ncks -O -C -D 3 -v var_nm.dot -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to
find variable
20130724: Verified problem still exists.
Stopped testing because inclusion of var_nm.dot broke all test scripts.
NB: Hard to fix since DAP interprets '.' as structure delimiter in
HTTP query string.
Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47
F. NOT YET FIXED (would require DAP protocol change)
Correctly read scalar characters over DAP.
DAP non-transparency: Works locally, fails through DAP server.
Problem, IMHO, is with DAP definition/protocol
Demonstration:
ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc
20120801: Verified problem still exists
Bug report not filed
Cause: DAP translates scalar characters into 64-element (this
dimension is user-configurable, but still...), NUL-terminated
strings so MD5 agreement fails
"Sticky" reminders:
A. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g.,
HDF4: AMSR MERRA MODIS ...
HDF5: GLAS ICESat Mabel SBUV ...
HDF-EOS5: AURA HIRDLS OMI ...
B. Pre-built executables for many OS's at:
http://nco.sf.net#bnr
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(
