The netCDF Operators NCO version 4.6.0 are ready.
http://nco.sf.net (Homepage, Mailing lists)
http://github.com/nco (Source Code, Releases, Developers)
What's new?
4.6.0 may be most notable for the debut of ncclimo, a new operator
that generates climatologies from monthly-mean input.
Perhaps it's a tie with ncap2, which has a singularly useful new
feature: variable lists/pointers. ncap2 also has a reduced memory
footprint and a function to simplify adding CF-bounds variables
(Thanks Henry!). As usual, ncremap continues to accrue useful
features, the most notable of which is learning grid information
from the CF "coordinates" attribute, if any.
Work on NCO 4.6.1 has commenced and will better support regridding
variables whose horizontal dimensions are not the
most-rapidly-varying.
Enjoy,
Charlie
NEW FEATURES (full details always in ChangeLog):
A. ncclimo produces climatological monthly means, seasonal means,
annual mean, and optionally regrids all these files. Like ncremap,
ncclimo is actually a front-end script to manage the complexity
of invoking the underlying operators (ncks for ncremap and ncra for
ncclimo). ncclimo produces climatologies starting from monthly-mean
files of most CESM-like model and observational data formats.
Invoke ncclimo with, e.g.,
ncclimo -s start_yr -e end_yr -c run_id -i drc_in -o drc_out
ncclimo -s 5 -e 10 -c hist -m ocn -i drc_in -o drc_out
ncclimo -s 1980 -e 2016 -c merra2_198001.nc -i drc_in -o drc_out
ncclimo has sophisticated parallelism and regridding options.
Datasets tested include ALM, CAM, CLM, CICE, CISM, CLM, MPAS-I,
MPAS-O, and POP. ncclimo resulted from work with the DOE ACME
project, and we are grateful for their support.
http://nco.sf.net/nco.html#ncclimo
http://nco.sf.net/nco.html#merra2
B. ncap2 now works with pointers to variables and attributes.
This allows, e.g., loops over arbitrary sets of variables, and
make creating aggregations of variables much simpler. An ncap2
script that converts all float variables to double precision:
@all=get_vars_in();
*sz=@all.size();
for(idx=0;idx<sz;idx++){
@var_nm=sprint(@all(idx));
if(*@var_nm.type() == NC_FLOAT) *@var_nm=*@var_nm.double();
}
Congratulations Henry for finishing this impressive functionality!
More documentation will be added in the coming weeks. For now,
http://nco.sf.net/nco.html#ncap2
C. ncap2 has reduced memory usage. Scalars are no longer stretched
to conform to arrays prior to arithmetic.
D. ncap2 has a new function to create CF-compliant bounds:
defdim("bnd_dmn",2);
bounds_var=make_bounds(crd_var,$bnd_dmn,"bounds_var_nm");
The "bounds_var_nm" is optional. Passing it causes make_bounds() to
add the attribute crd_var@bounds="bounds_var_nm".
http://nco.sf.net/nco.html#make_bounds
E. ncap2 now propagates metadata when Left-Hand-Casting variables.
Previous versions did not propagate metadata to LHC variables.
This fix makes converting the _type_ of a variable easier.
For example, to change one_dmn_rec_var to NC_SHORT, use
ncap2 -s 'one_dmn_rec_var[time]=short(one_dmn_rec_var)' in.nc out.nc
The output file will contain the original metadata.
http://nco.sf.net/nco.html#ncap2
F. ncatted new mode "nappend" appends only to existing attributes.
If the attribute does not yet exist, it is not created.
Same as "append" mode, except "append" will create attributes.
ncatted -a long_name,T,n,c,' nappended text' in.nc
http://nco.sf.net/nco.html#ncatted
G. ncpdq now has a third unpacking mode.
Access non-default modes with --upk=[0,1,2] where
Mode 0: Applies netCDF convention. Default mode:
upk=scale_factor*pck+add_offset
Mode 1: HDF4 MODIS MOD08 convention:
upk=scale_factor*(pck-add_offset)
Mode 2: HDF4 MODIS MOD13 convention:
upk=(pck-add_offset)/scale_factor
Yes, be careful unpacking HDF4 data!
http://nco.sf.net/nco.html#hdf_upk
H. ncra can now honor the CF convention for climatology bounds.
Currently this is opt-in with the --cb or --c2b switches.
--cb causes ncra to:
1. Add a "climatology" attribute with value "climatology_bounds" to
the time coordinate, if necessary
2. Remove the "bounds" from the time coordinate, if necessary
3. Output a variable named "climatology_bounds" with values that
are minima/maxima of the input time coordinate bounds
4. Omit any input time coordinate bounds attribute and variable
5. Ensure cell_methods attribute for all variables is appropriate
--c2b is like --cb except --c2b converts the input "climatology"
bounds to a non-climatology "bounds" in the output.
Use --c2b when averaging sub-sampled climatologies to produce a
continuous (non-climatologically sub-sampled) mean.
ncra --cb *_01.nc clm_JAN.nc
ncra --cb clm_DEC.nc clm_JAN.nc clm_FEB.nc clm_DJF.nc
ncra --c2b clm_DJF.nc clm_MAM.nc clm_JJA.nc clm_SON.nc clm_ANN.nc
Depending on feedback, we may make --cb the default
http://nco.sf.net/nco.html#cb
I. ncremap now defaults to $TMPDIR for storing intermediate files.
This is user-configurable with the -U switch.
http://nco.sf.net/nco.html#ncremap
J. ncks can now ingest and de-interleave ENVI images in BIL, BSQ, and
BIP formats and store them as any netCDF type. ncks is a viable
(and faster) replacement for that small subset of tasks normally
done by the venerable GDAL toolkit (gdal_translate in particular).
Thanks to David LeBauer of the University of Illinois/NCSA and DOE
TERRAREF for supporting this feature.
ncks --trr_wxy=926,1600,1 --trr typ_in=NC_USHORT --trr ntl_in=bil \
--trr_in=in.nc foo.nc out.nc
http://nco.sf.net/nco.html#terraref
K. NCO now treats as associated coordinates all variables listed in
"grid_mapping" attributes. Variables such as
"char albers_conical_equal_area" are automatically extracted along
with variables that them. ncecat will not wrap mappings in a record
dimension.
http://nco.sf.net/nco.html#grid_mapping
L. ncremap and ncks now follow the CF "coordinates" convention when
inferring grids and remapping. One specifies the variable that may
have the "coordinates" attribute, and NCO will inspect and apply
the "coordinates" convention for that variable for grid generation
and remapping.
ncremap -V var_nm -s src.nc -d dst.nc -m map.nc
ncremap -V var_nm --map map.nc in.nc out.nc
ncks --rgr_var=var_nm --rgr nfr=y --rgr grid=grd.nc in.nc foo.nc
ncks --rgr_var=var_nm --map map.nc in.nc out.nc
http://nco.sf.net/nco.html#ncremap
http://nco.sf.net/nco.html#regrid
BUG FIXES:
A. Correct ncatted documentation for "append" mode.
B. Fix Append mode to follow same code path and defaults when
requested with -A and with interactive responses to the
exit/append/overwrite question. Thanks to Parker Norton for
pointing out this inconsistency.
C. ncpdq fix complex reordering in group hierarchies.
Previously ncpdq could misorder dimensions when multiple
different dimensions shared the same short name (in different
groups). Thanks to Pedro Vicente for tracking-down and fixing
this bug! And finding and fixing another ncpdq bug that had
not yet been reported in the wild.
D. ncap2 fix bug propagating attributes of variables with whitespace
in their names.
KNOWN PROBLEMS DUE TO NCO:
This section of ANNOUNCE reports and reminds users of the
existence and severity of known, not yet fixed, problems.
These problems occur with NCO 4.6.0 built/tested under
MacOS with netCDF 4.3.3.1 on HDF5 1.8.16 and with
Linux with netCDF 4.4.1-development (20160212) on HDF5 1.8.13.
A. NOT YET FIXED (NCO problem)
Correctly read arrays of NC_STRING with embedded delimiters in
ncatted arguments
Demonstration:
ncatted -D 5 -O -a
new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc
~/foo.nc
ncks -m -C -v att_var ~/foo.nc
20130724: Verified problem still exists
TODO nco1102
Cause: NCO parsing of ncatted arguments is not sophisticated
enough to handle arrays of NC_STRINGS with embedded delimiters.
B. NOT YET FIXED (NCO problem?)
ncra/ncrcat (not ncks) hyperslabbing can fail on variables with
multiple record dimensions
Demonstration:
ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc
20140826: Verified problem still exists
20140619: Problem reported by rmla
Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
Workaround: Convert to fixed dimensions then hyperslab
KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:
A. NOT YET FIXED (netCDF4 or HDF5 problem?)
Specifying strided hyperslab on large netCDF4 datasets leads
to slowdown or failure with recent netCDF versions.
Demonstration with NCO <= 4.4.5:
time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
Demonstration with NCL:
time ncl < ~/nco/data/ncl.ncl
20140718: Problem reported by Parker Norton
20140826: Verified problem still exists
20140930: Finish NCO workaround for problem
Cause: Slow algorithm in nc_var_gets()?
Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
Workaround #2: Convert file to netCDF3 first, then use stride
B. NOT YET FIXED (netCDF4 library bug)
Simultaneously renaming multiple dimensions in netCDF4 file can
corrupt output
Demonstration:
ncrename -O -d lev,z -d lat,y -d lon,x ~/nco/data/in_grp.nc ~/foo.nc
# Completes but file is unreadable
ncks -v one ~/foo.nc
20150922: Confirmed problem reported by Isabelle Dast, reported to
Unidata
20150924: Unidata confirmed problem
20160212: Verified problem still exists in netCDF library
20160512: Ditto
Bug tracking: https://www.unidata.ucar.edu/jira/browse/fxm
More details: http://nco.sf.net/nco.html#ncrename_crd
C. NOT YET FIXED (would require DAP protocol change?)
Unable to retrieve contents of variables including period '.' in name
Periods are legal characters in netCDF variable names.
Metadata are returned successfully, data are not.
DAP non-transparency: Works locally, fails through DAP server.
Demonstration:
ncks -O -C -D 3 -v var_nm.dot -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to
find variable
20130724: Verified problem still exists.
Stopped testing because inclusion of var_nm.dot broke all test scripts.
NB: Hard to fix since DAP interprets '.' as structure delimiter in
HTTP query string.
Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47
D. NOT YET FIXED (would require DAP protocol change)
Correctly read scalar characters over DAP.
DAP non-transparency: Works locally, fails through DAP server.
Problem, IMHO, is with DAP definition/protocol
Demonstration:
ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc
20120801: Verified problem still exists
Bug report not filed
Cause: DAP translates scalar characters into 64-element (this
dimension is user-configurable, but still...), NUL-terminated
strings so MD5 agreement fails
"Sticky" reminders:
A. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g.,
HDF4: AMSR MERRA MODIS ...
HDF5: GLAS ICESat Mabel SBUV ...
HDF-EOS5: AURA HIRDLS OMI ...
B. Pre-built executables for many OS's at:
http://nco.sf.net#bnr
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(
