The netCDF operators NCO version 4.1.0 are ready.
http://nco.sf.net (Homepage)
http://dust.ess.uci.edu/nco (Homepage "mirror")
This release implements NCO's first data security/integrity checks,
consisting of MD5 hashes of data for each variable. These optional
checks are available in ncks and ncrcat. Thanks to Patrick from IPSL
for suggesting this.
A new "feature" of netCDF3 has been identified that can lead to
unusually slow performance. Unfortunately, this problem is triggered
by an aggregate pattern of file access so the workaround must be
implemented in the application software (i.e., NCO) rather than in the
netCDF library itself. It is normally encountered on Multi-record
Multi-variable netCDF3 files (MM3s), so we call this the
MM3-workaround. If you encounter unusually slow NCO performance while
using NCO to analyze MM3 files on a large blocksize filesystem,
chances are you are encountering an MM3-induced slowdown.
This NCO release implements the MM3-workaround for ncks. It speeds-up
common ncks sub-setting on NCAR's GLADE by 10-50x. MM3-induced
slowdowns may be present in other NCO operators and we are
prioritizing our MM3-patches to those encountered most often. Thanks
to Gary Strand for reporting this problem, and to Russ Rew for
creating the workaround algorithm, which is also now in nccopy.
More on this story as it develops at
http://nco.sf.net#bug_mm3
NASA has confirmed funding NCO to develop support for netCDF4 groups
and to write NCO wrappers for HDF-EOS files. A job ad has been posted
for a programmer position associated with this project:
http://dust.ess.uci.edu/hire/prg_anl_05_adv.pdf
http://ess.uci.edu/opportunity/zender20120327
Work on NCO 4.1.1 is underway, focused on making a more robust
build/configuration environment, addressing the MM3 issue in other
(non-ncks) operators, and on simplifying issues described in the
KNOWN BUGS NOT YET FIXED section below.
Enjoy,
Charlie
Other "New stuff" in 4.1.0 summary (full details always in ChangeLog):
A. Workaround slow copies of record variables.
Normally encountered on Multi-record Multi-variable netCDF3 files
(MM3s), this is also known as the MM3-workaround.
It speeds-up common ncks sub-setting on NCAR's GLADE by 10-50x.
Thanks to Gary Strand for reporting this problem, and to Russ Rew
for creating the workaround algorithm, which is also now in nccopy.
http://nco.sf.net#bug_mm3
B. MD5 digest support. Digests can help ensure data integrity.
They will alert, e.g., some ocurrances of the NC_NOFILL bug.
However, the execution time cost can be significant, requiring
two MD5 digests and an extra-from-disk for all data.
ncks -O -D 1 -H -C --md5 -v md5_a,md5_abc ~/nco/data/in.nc
ncrcat -O -D 1 --md5 -p ~/nco/data in.nc in.nc ~/foo.nc
http://nco.sf.net/nco.html#md5
C. Expand and redefine ncap2 treatment of sorting and maps:
var_out=sort(var_in,&srt_map); // Ascending sort
var_out=asort(var_in,&srt_map); // Accending sort
var_out=dsort(var_in,&srt_map); // Desending sort
var_out=remap(var_in,srt_map); // Apply srt_map to var_in
var_out=unmap(var_in,srt_map); // Reverse what srt_map did to var_in
dsr_map=invert_map(srt_map); // Produce "de-sort" map that inverts
srt_map
http://nco.sf.net/nco.html#sort
Thanks to Henry Butowsky for this functionality
D. Fix bug with ncap2 print() command. Thanks to Henry Butowsky.
E. ncrename/ncatted succeed when output file is specified and input
file is read-only. Formerly this failed:
chmod 444 ~/foo.nc;ncrename -O -v one,uno ~/foo.nc ~/foo2.nc
Now NCO executes chmod() so it succeeds. Ditto for ncatted.
Thanks to Paida for reporting this.
F. Fix bug where ncap2 could fail when averaging degenerate arrays.
Thanks to Henry Butowsky.
G. Allow ncatted to change missing value's to/from IEEE NaNs
With this feature, one can now perform arithmetic on variables that
use NaN for _FillValue simply by changing NaN to a normal number,
then performing arithmetic, then changing back:
ncatted -O -D 3 -a _FillValue,nan_arr,o,f,1.0e36 # Change from NaN
ncwa, ncra, etc. # Perform any arithmetic
ncatted -O -D 3 -a _FillValue,fll_val,o,f,nan # Change to NaN
KNOWN BUGS NOT YET FIXED:
This section of the ANNOUNCE file is intended to make clear the
existence and severity of known, not yet fixed, problems.
A. NOT YET FIXED
Correctly read netCDF4 input over DAP, write netCDF4 output, then
read resulting file.
Replacing netCDF4 with netCDF3 in either location of preceding
sentence leads to success.
DAP non-transparency: Works locally, fails through DAP server.
Demonstration:
ncks -4 -O -v three_dmn_rec_var
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods/in_4.nc ~/foo.nc
ncks ~/foo.nc # breaks with "NetCDF: Invalid dimension ID or name"
20120329: Verified problem still exists
Bug report filed: netCDF #QUN-641037: dimension ID ordering assumptions
B. NOT YET FIXED
netCDF4 library fails when renaming dimension and variable using
that dimension, in either order. Works fine with netCDF3.
Problem with netCDF4 library implementation.
Demonstration:
ncks -O -4 -v lat_T42 ~/nco/data/in.nc ~/foo.nc
ncrename -O -D 2 -d lat_T42,lat -v lat_T42,lat ~/foo.nc ~/foo2.nc #
Breaks with "NetCDF: HDF error"
ncks -m ~/foo.nc
20120329: Verified problem still exists
Bug report filed: netCDF #YQN-334036: problem renaming dimension and
coordinate in netCDF4 file
C. NOT YET FIXED
Unable to retrieve contents of variables with period '.' in name
Metadata is returned successfully, data is not.
DAP non-transparency: Works locally, fails through DAP server.
Demonstration:
ncks -O -C -D 3 -v var_nm.dot -p
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods in.nc # Fails to
find variable
20120329: Verified problem still exists. Stopped testing because
inclusion of var_nm.dot broke all test scripts.
Bug report filed: https://www.unidata.ucar.edu/jira/browse/NCF-47
"Sticky" reminders:
A. All operators support netCDF4 chunking options:
These options can improve performance on large datasets.
Large file users: Send us suggestions on useful chunking patterns!
ncks -O -4 --cnk_plc=all in.nc out.nc
http://nco.sf.net/nco.html#chunking
B. Pre-built, up-to-date Debian Sid & Ubuntu Maverick packages:
http://nco.sf.net#debian
C. Pre-built Fedora and CentOS RPMs:
http://nco.sf.net#rpm
D. Did you try SWAMP (Script Workflow Analysis for MultiProcessing)?
SWAMP efficiently schedules/executes NCO scripts on remote servers:
http://swamp.googlecode.com
SWAMP can work command-line operator analysis scripts besides NCO.
If you must transfer lots of data from a server to your client
before you analyze it, then SWAMP will likely speed things up.
E. NCO support for netCDF4 features is tracked at
http://nco.sf.net/nco.html#nco4
NCO supports netCDF4 atomic data types, compression, and chunking.
NCO 4.1.0 with was built and tested with HDF5 hdf5-1.8.7 and with
4.2-snapshot2012020522.
NCO may not build with earlier, and should build with later, netCDF4
releases.
This is particularly true since NCO 4.1.0 takes advantage of an
internal change to the netCDF nc_def_var_chunking() API in June 2009.
export NETCDF4_ROOT=/usr/local/netcdf4 # Set netCDF4 location
cd ~/nco;./configure --enable-netcdf4 # Configure mechanism -or-
cd ~/nco/bld;make dir;make all;make ncap2 # Old Makefile mechanism
F. Have you seen the NCO logo candidates by Tony Freeman, Rich
Signell, Rob Hetland, and Andrea Cimatoribus?
http://nco.sf.net
Tell us what you think...
--
Charlie Zender, Department of Earth System Science
University of California, Irvine 949-891-2429 )'(
