The netCDF operators NCO version 4.2.1 are ready.
http://nco.sf.net (Homepage)
http://dust.ess.uci.edu/nco (Homepage "mirror")
This release is laden with hyperslabbing and performance features.
All hyperslabbing operators can now count backwards.
Time-consuming intermediate files can be bypassed.
ncra and ncrcat handle sub-groups of records at regular intervals.
Time re-basing now works on "bounds" variables, e.g., time_bnds.
Together, these features significantly ease working with large
datasets like CMIP5.
Also NCO supports the new "diskless" feature of netCDF 4.2.1.
Autoconf builds are now regularly tested on more platforms.
Keep the suggestions coming for features you want.
Work on NCO 4.2.2 is underway. Expect ease-of-build improvements, an
optimization for diskless/RAM files, and better netCDF4 support.
Enjoy,
Charlie
Other "New stuff" in 4.2.1 summary (full details always in ChangeLog):
NEW FEATURES:
A. Duration/Sub-cycles in Multi-Record Operators (MROs). Huh?
In plainer English, ncra and ncrcat now handle a fourth hyperslab
parameter that specifies the number of consecutive records over
which the operation should act. Now MROs can, say, stride along the
time dimension by leaps of twelve indices, averaging three
consecutive records at a time. Yes, like a seasonal average!
Not only that, but ncra can be instructed to output its operations
(e.g., averages) every sub-cycle, so that, e.g., interannual
timeseries of seasonal averages can be created in one fell swoop.
This may be the most powerful new NCO feature in many moons.
An extended example is described in the manual at
http://nco.sf.net/nco.html#drn
B. Bypass intermediate temporary files
ncks --no_tmp_fl ~/nco/data/in.nc ~/foo.nc
Since 2004 at least four people, including Remik Ziemlinski, who I
believe was first, have requested this feature, and now it is here.
http://nco.sf.net/nco.html#no_tmp_fl
C. One may now extract extract the last N elements of a hyperslab.
Negative integers as min or max elements of hyperslab specifications
indicate offsets from the end (Python also uses this convention).
Previously, for example, -d time,-2,-1 caused a domain error.
Now it means select the second-to-last and penultimate timesteps.
Negative integers work for min and mix indices but not for stride.
Thanks to Nick Bower for requesting this feature (in 2008!).
http://nco.sf.net/nco.html#hyp
D. Diskless files aka RAM-disk files
This feature requires the newest netCDF library, version 4.2.1.
ncks --ram ~/nco/data/in.nc ~/foo.nc
http://nco.sf.net/nco.html#ram
E. Re-basing bounds of record coordinates
Time rebasing is invoked when numerous files share a common record
coordinate, and the record coordinate units change among input files.
Usually this occurs when the recoordinate is a time-based variable,
and times are recorded in units of a time-since-basetime, and the
basetime changes from file to file.
Now NCO automatically rebases not only the record coordinate (e.g.,
time) but also any bounds associated with the record coordinate
(i.e., time_bnds). Thanks to Gerry Potter for suggesting this.
http://nco.sf.net/nco.html#rbs
F. Windows builds
Eliminated some behavioral differences on Windows vs. UNIX NCO.
Windows NCO now uses temporary files by default, like UNIX.
Windows NCO still lacks networking capabilities, e.g., scp/sftp.
http://nco.sf.net/nco.html#windows
G. Building NCO using the cross-platform Qt framework improved.
Now works on Ubuntu Linux using default .deb-based software stack.
Qt Unix build now includes ncwa.
BUG FIXES:
A. When invoked for min/max operations, ncra/ncea now computes means
of coordinate variables, as documented. Thanks to user Roux for
reporting this issue and supplying the patch.
A. Enable --fl_fmt switch synonym to --file_format on all applicable
operators. The operators now conform with the documentation.
A. ncpdq -U now unpacks coordinate variables. ncpdq does not, by
default, pack coordinate variables for reasons explained in the
manual. Yet NCO may be used on files that were packed by other
software and then it needs to unpack coordinate variables so...
http://nco.sf.net/nco.html#ncpdq
A. ncra/ncrcat averaging/concatenation of record coordinate-bounded
variable across two non-monotonic files now selects correct records
in all cases.
KNOWN BUGS NOT YET FIXED:
This section of the ANNOUNCE file reports and reminds users of the
existence and severity of known, not yet fixed, problems.
These problems occur with NCO 4.2.1 built/tested with netCDF
4.2.1 on top of HDF5 hdf5-1.8.9 with these methods:
cd ~/nco;./configure --enable-netcdf4 # Configure mechanism -or-
cd ~/nco/bld;make dir;make all;make ncap2 # Old Makefile mechanism
A. NOT YET FIXED
Correctly read netCDF4 input over DAP, write netCDF4 output, then
read resulting file.
Replacing netCDF4 with netCDF3 in either location of preceding
sentence leads to success.
DAP non-transparency: Works locally, fails through DAP server.
Demonstration:
ncks -4 -O -v three_dmn_rec_var
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods/in_4.nc ~/foo.nc
ncks ~/foo.nc # breaks with "NetCDF: Invalid dimension ID or name"
20120731: Verified problem still exists
Bug report filed: netCDF #QUN-641037: dimension ID ordering assumptions
B. NOT YET FIXED
netCDF4 library fails when renaming dimension and variable using
that dimension, in either order. Works fine with netCDF3.
Problem with netCDF4 library implementation.
Demonstration:
ncks -O -4 -v lat_T42 ~/nco/data/in.nc ~/foo.nc
ncrename -O -D 2 -d lat_T42,lat -v lat_T42,lat ~/foo.nc ~/foo2.nc #
Breaks with "NetCDF: HDF error"
ncks -m ~/foo.nc
20120731: Verified problem still exists
Bug report filed: netCDF #YQN-334036: problem renaming dimension and
coordinate in netCDF4 file
C. NOT YET FIXED
Unable to retrieve contents of variables including period '.' in name
Metadata is returned successfully, data is not.
DAP non-transparency: Works locally, fails through DAP server.
Demonstration:
ncks -O -C -D 3 -v var_nm.dot -p
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods in.nc # Fails to
find variable
20120731: Verified problem still exists.
Stopped testing because inclusion of var_nm.dot broke all test scripts.
NB: Problem hard to fix since DAP interprets '.' as structure
delimiter in query string of HTTP requests.
Bug report filed: https://www.unidata.ucar.edu/jira/browse/NCF-47
D. NOT YET FIXED
Correctly read scalar characters over.
DAP non-transparency: Works locally, fails through DAP server.
Demonstration:
ncks -O -D 1 -H -C -m --md5 -v md5_a -p
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods in.nc
20120801: Verified problem still exists
Bug report not yet filed
"Sticky" reminders:
A. Pre-built, up-to-date Debian Sid & Ubuntu packages:
http://nco.sf.net#debian
B. Pre-built Fedora and CentOS RPMs:
http://nco.sf.net#rpm
C. Pre-built Windows (native) and Cygwin binaries:
http://nco.sf.net#windows
D. Pre-built AIX binaries:
http://nco.sf.net#aix
E. Did you try SWAMP (Script Workflow Analysis for MultiProcessing)?
SWAMP efficiently schedules/executes NCO scripts on remote servers:
http://swamp.googlecode.com
SWAMP can work command-line operator analysis scripts besides NCO.
If you must transfer lots of data from a server to your client
before you analyze it, then SWAMP will likely speed things up.
F. NCO support for netCDF4 features is tracked at
http://nco.sf.net/nco.html#nco4
NCO supports netCDF4 atomic data types, compression, and chunking.
G. Have you seen the NCO logo candidates by Tony Freeman, Rich
Signell, Rob Hetland, and Andrea Cimatoribus?
http://nco.sf.net
Tell us what you think...
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(
