The netCDF Operators NCO version 4.3.4 are ready.
http://nco.sf.net (Homepage)
http://dust.ess.uci.edu/nco (Homepage "mirror")
Ooops. The last two releases of ncpdq had broken unpacking.
It unpacked variables correctly, but inadvertently stored the results
with the original packing attributes. Successive operations would
therefore unpack the variable twice, leading to incorrect results.
This annoys us enough to relase 4.3.4 early, before converting any
more operators to work with groups. Upgrade recommended if you ever
use ncpdq to unpack.
There are also a few new features: including more legible CDL printing
of strings, and a new switch that overrides the default netCDF
unpacking algorithm with the HDF unpacking algorithm.
Multi-slabbing was introduced to NCO by Henry Butowsky in 2002 and
I've never fully understood its capabilities. I learned a lot
more about MSA in the last few weeks, and have now documented it.
Read the docs referenced below and you too may be surprised at how
easy MSA makes complex dataset manipulation like rotating grids.
Work on NCO 4.3.5 is underway and includes improved netCDF4 support
for more NCO operators (ncatted, ncea, ncra, ncrcat) and improved
support for HDF4 files.
Enjoy,
Charlie
"New stuff" in 4.3.4 summary (full details always in ChangeLog):
NEW FEATURES:
A. Improve CDL legibility of NC_CHAR attributes with embedded newlines
http://nco.sf.net/nco.html#cdl
B. Global --hdf_upk switch implements HDF unpacking convention:
HDF unpack definition: unpacked=scale_factor*(packed-add_offset).
netCDF unpack definition: unpacked=(scale_factor*packed)+add_offset.
See the difference?
NCO automatically unpacks data before performing arithmetic, and by
default NCO uses the netCDF convention. Data originally from an HDF
file (e.g., NASA EOS) may well be packed with the HDF convention
and must be unpacked with the same convention. Users must tell NCO
when to use the HDF convention instead of the netCDF convention.
This is what we in the bizness call an interoperability issue.
Know what are doing when you use this switch. The wrong algorithm
will hopefully produce _noticeably_ bad results, but maybe not.
ncl_convert2nc modis_1.hdf modis_2.hdf modis_200*.hdf
ncbo --hdf_upk modis_1.nc modis_2.nc diff.nc
ncra --hdf_upk modis_200*.nc modis_avg.nc
ncwa --hdf_upk modis_1.hdf modis_avg.nc
ncpdq --hdf_upk -P xst_new modis_1.nc modis_1.nc
http://nco.sf.net/nco.html#upk
http://nco.sf.net/nco.html#hdf_upk
http://nco.sf.net/nco.html#ncpdq
C. Multi-slab Algorithm user-ordering with --msa is now documented.
This eases data re-ordering. A fully documented example uses MSA to
rotate from a [-180,180) to a [0,360) longitude grid in two steps:
ncks -O --msa -d Lon,0.,180. -d Lon,-180.,-1.0 in.nc out.nc
ncap2 -O -s 'where(Lon < 0) Lon=Lon+360' out.nc out.nc
http://nco.sf.net/nco.html#msa
http://nco.sf.net/nco.html#msa_usr_rdr
BUG FIXES:
A. Fix ncpdq unpacking.
Versions 4.3.2-4.3.3 of ncpdq did not correctly store unpacked
variables. These versions unpacked (when specified with -U or -P
upk) the values, but inadvertently stored their original packing
attributes with the unpacked values. This could lead further
operators to assume that the values were still packed. Hence
consecutive operations could lead to incorrect values.
All ncpdq users are encouraged to upgrade.
NB: this bug did not affect the other arithmetic operators which
unpack data prior to arithmetic.
http://nco.sf.net/nco.html#bug_ncpdq_upk
KNOWN BUGS NOT YET FIXED:
This section of ANNOUNCE reports and reminds users of the
existence and severity of known, not yet fixed, problems.
These problems occur with NCO 4.3.4 built/tested with netCDF
4.3.0 on top of HDF5 hdf5-1.8.9 with these methods:
cd ~/nco;./configure --enable-netcdf4 # Configure mechanism -or-
cd ~/nco/bld;make dir;make all;make ncap2 # Old Makefile mechanism
A. NOT YET FIXED
Correctly read netCDF4 input over DAP, write netCDF4 output, then
read resulting file.
Replacing netCDF4 with netCDF3 in either location of preceding
sentence leads to success.
DAP non-transparency: Works locally, fails through DAP server.
Unclear whether resulting file is "legal" because of dimension ID
ordering assumptions.
Demonstration:
ncks -4 -O -v three_dmn_rec_var
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods/in_4.nc ~/foo.nc
ncks ~/foo.nc # breaks with "NetCDF: Invalid dimension ID or name"
20130724: Unable to verify since in_4.nc no longer accessible on
Unidata DAP server
Bug report filed: netCDF #QUN-641037: dimension ID ordering assumptions
B. NOT YET FIXED
netCDF4 library fails when renaming dimension and variable using
that dimension, in either order. Works fine with netCDF3.
Problem with netCDF4 library implementation.
Demonstration:
ncks -O -4 -v lat_T42 ~/nco/data/in.nc ~/foo.nc
ncrename -O -D 2 -d lat_T42,lat -v lat_T42,lat ~/foo.nc ~/foo2.nc #
Breaks with "NetCDF: HDF error"
ncks -m ~/foo.nc
20130724: Verified problem still exists
Bug report filed: netCDF #YQN-334036: problem renaming dimension and
coordinate in netCDF4 file
C. NOT YET FIXED (requires change to DAP protocol)
Unable to retrieve contents of variables including period '.' in name
Periods are legal characters in netCDF variable names.
Metadata are returned successfully, data are not.
DAP non-transparency: Works locally, fails through DAP server.
Demonstration:
ncks -O -C -D 3 -v var_nm.dot -p
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods in.nc # Fails to
find variable
20130724: Verified problem still exists.
Stopped testing because inclusion of var_nm.dot broke all test scripts.
NB: Hard to fix since DAP interprets '.' as structure delimiter in
HTTP query string.
Bug report filed: https://www.unidata.ucar.edu/jira/browse/NCF-47
D. NOT YET FIXED (requires change to DAP protocol)
Correctly read scalar characters over DAP.
DAP non-transparency: Works locally, fails through DAP server.
Problem, IMHO, is with DAP definition/protocol
Demonstration:
ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p
http://motherlode.ucar.edu:8080/thredds/dodsC/testdods in.nc
20120801: Verified problem still exists
Bug report not filed
Cause: DAP translates scalar characters into 64-element (this
dimension is user-configurable, but still...), NUL-terminated
strings so MD5 agreement fails
E. NOT YET FIXED (NCO problem)
Correctly read arrays of NC_STRING with embedded delimiters in
ncatted arguments
Demonstration:
ncatted -D 5 -O -a
new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc
~/foo.nc
20130724: Verified problem still exists
TODO nco1102
Cause: NCO parsing of ncatted arguments is not yet sophisticated
enough to handle arrays of NC_STRINGS with embedded delimiters.
"Sticky" reminders:
A. Pre-built, up-to-date Debian Sid & Ubuntu packages:
http://nco.sf.net#debian
B. Pre-built Fedora and CentOS RPMs:
http://nco.sf.net#rpm
C. Pre-built Windows (native) and Cygwin binaries:
http://nco.sf.net#windows
D. Pre-built AIX binaries:
http://nco.sf.net#aix
E. Did you try SWAMP (Script Workflow Analysis for MultiProcessing)?
SWAMP efficiently schedules/executes NCO scripts on remote servers:
http://swamp.googlecode.com
SWAMP can work command-line operator analysis scripts besides NCO.
If you must transfer lots of data from a server to your client
before you analyze it, then SWAMP will likely speed things up.
F. NCO support for netCDF4 features is tracked at
http://nco.sf.net/nco.html#nco4
NCO supports netCDF4 atomic data types, compression, chunking, and
groups.
G. Reminder that ncks, ncecat, ncbo, ncflint, and ncpdq work on many common
HDF5 datasets, e.g.,
NASA AURA HIRDLS HDF-EOS5
NASA ICESat GLAS HDF5
NASA SBUV HDF5...
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(
