The netCDF operators NCO version 4.0.4 are ready.
http://nco.sf.net (Homepage)
http://dust.ess.uci.edu/nco (Homepage "mirror")
This version contains several improvements and a few fixes. New stuff:
Chunking is now supported on all relevant operators, i.e., all except
ncrename and ncatted. ncap2, moreover, now propagates deflation, and
tries to propagate chunking options, through derived expressions.
ncpdq now tries to avoid packing grid properties like coordinates.
Metadata printed by ncks is now re-formatted, expanded, and simpler.
Scripts that depend on the exact format of ncks output may break.
Let us know if this causes you major heartburn and, if so, we will
consider implementing an option to restore the previous format.
Work on NCO 4.0.5 is underway. Areas of improvement include
support for spectral transforms, more chunking rulesets.
"New stuff" in 4.0.4 details:
A. Chunking supported in all operators.
The exception is ncap2, which propagates chunking from fixed
and copied variables, though not through algebraic expressions.
(This capability may be in the next version.)
http://nco.sf.net/nco.html#cnk
B. ncecat, and all other operators, now retain the input deflation
level, if any, for a variable, as the default for the output file.
C. Deflation level (--dfl_lvl) propagates into ncap2 derived variables
Thanks to Henry Butowsky.
D. ncks -m now always summarizes metadata on chunking, compression, and
packing
Moreover, it prints metadata with clearer English descriptions
E. ncpdq no longer packs grid properties such as coordinate variables,
CF "special variables, or weights. Rationale: Coordinate variables
are 1D, so packing saves little space, and the resulting files are
difficult for humans to read. Similarly, Gaussian and area weights
are often used to multiply other variables in re-inflated
(unpacked) files. Packing such weights causes a considerable loss
of precision in the result. If users wish to pack such variables,
we could implement new packing policies that explicitly packs
coordinates and weighting variables. We welcome your feedback. A
workaround for those needing to pack such variables now, is to use
the packing functions in ncap2, or to rename the grid properties
prior to calling ncpdq.
F. Stricter checks to identify malformed user-input strings.
Issue more informative warnings, hints, and errors when detected.
G. Prior versions of ncrename -a .old_nm,new_nm in.nc out.nc
would, if old_nm did not exist, write the correct file then exit
with an error message although no error had occurred.
http://nco.sf.net#bug_ncrename_dot
H. MacIntosh users who do not use MacPorts can install NCO binaries
by clicking on an Apple Disk Image (.dmg) file.
Thanks to Chad Cantwell of UCI.
http://nco.sf.net#dmg
"Sticky" reminders:
J. ncks and ncecat support netCDF4 chunking options:
These options can improve performance on large datasets.
Large file users: Send us suggestions on useful chunking patterns!
More useful chunking patterns may be implemented in NCO 4.0.4.
ncks -O -4 --cnk_plc=all in.nc out.nc
http://nco.sf.net/nco.html#chunking
K. Pre-built, up-to-date Debian Sid & Ubuntu Karmic packages:
http://nco.sf.net#debian
L. Pre-built Fedora and CentOS RPMs:
http://nco.sf.net#rpm
M. Did you try SWAMP (Script Workflow Analysis for MultiProcessing)?
SWAMP efficiently schedules/executes NCO scripts on remote servers:
http://swamp.googlecode.com
SWAMP can work command-line operator analysis scripts besides NCO.
If you must transfer lots of data from a server to your client
before you analyze it, then SWAMP will likely speed things up.
N. NCO support for netCDF4 features is tracked at
http://nco.sf.net/nco.html#nco4
NCO supports netCDF4 atomic data types, compression, and chunking.
NCO 4.0.4 with was built and tested with HDF5 hdf5-1.8.4-patch1 and with
netCDF4 4.1.2-beta1-snapshot2010080820
NCO may not build with earlier, and should build with later, netCDF4
releases.
This is particularly true since NCO 4.0.4 takes advantage of an
internal change to the netCDF nc_def_var_chunking() API in June 2009.
export NETCDF4_ROOT=/usr/local/netcdf4 # Set netCDF4 location
cd ~/nco;./configure --enable-netcdf4 # Configure mechanism -or-
cd ~/nco/bld;./make NETCDF4=Y allinone # Old Makefile mechanism
O. Have you seen the NCO logo candidates by Tony Freeman, Rich
Signell, Rob Hetland, and Andrea Cimatoribus?
http://nco.sf.net
Tell us what you think...
Enjoy,
Charlie
GSL functions added to ncap2 in 4.0.4:
none!
(Most desired GSL functions have been merged)
--
Charlie Zender, Department of Earth System Science
University of California, Irvine (949) 891-2429 :)
