The netCDF Operators NCO version 4.4.6 are ready.
http://nco.sf.net (Homepage)
http://dust.ess.uci.edu/nco (Homepage "mirror")
This is a bugfix, stability, and performance release.
It fixes an ncra bug that affects packed records with missing values,
and it fixes an ncwa bug that affects minimization and maximization
operations on packed variables. Affected users would likely have
noticed being bit by these bugs since the answers generated were,
well, noticeable. For these reasons alone, we recommend that users of
packed datasets upgrade to 4.4.6.
We rewrote ncrename to enhance its group-oriented features.
We also implemented a performance optimization that dramatically
improves access times to strided hyperslabs in netCDF4 datasets.
Minor improvements include new ncks printing and hyperslab options,
support for HDF4 chunking/compression, and ncap2 argument passing.
Work on NCO 4.4.7 is underway, still focused on stability and speed.
This includes more netCDF4 mop-up and new chunking features.
Enjoy,
Charlie
NEW FEATURES (full details always in ChangeLog):
A. ncrename expanded functionality on group-oriented files:
ncrename now supports group attributes and variable attributes with
the same syntax, provides distinct options to change only global
metatdata or metadata in every group, and works with multiple
instances of the same object, e.g., multiple instances of the same
variable name in different groups.
ncrename -a g1@old_att,new_att in.nc # Rename in g1 only
ncrename -a group@old_att,new_att in.nc # Rename in all groups
ncrename -a global@old_att,new_att in.nc # Rename in root only
http://nco.sf.net/nco.html#ncrename
B. Optimization of strided hyperslab access to netCDF4 data:
Multiple users complain that access to strided hyperslabs of
netCDF4 datasets is orders of magnitude slower than expected.
This occurs with other languages (like NCL) and appears to be
caused by lower-level libraries. Until the problem is solved at the
root, we have developed and implemented a transparent workaround
that fixes the issue for the common case where strides are taken in
only one dimension, e.g., -d time,0,,12. Strided hyperslabs like
ncks -O -d time,0,,12 ~/in.nc4 ~/foo.nc
now complete in nearly the same amount of time as non-strided.
This workaround works transparently with any version of netCDF.
http://nco.sf.net#bug_nc4_srd
C. ncap2 command-line and script arguments can now both be specified
AND executed. Command-line args are executed before script:
ncap2 -v -s "lon=${longitude}" -S in_4.nc out.nc
This is useful for passing shell variables into NCO scripts.
Pass the shell variables with -s and the -S scripts will use them.
Thanks to Henry Butowsky for this.
http://nco.sf.net/nco.html#ncap2
D. ncks -V = --var_val is new switch to print only variable values.
Compare and contrast -Q, -V, and --no_nm_prn:
% ncks -v three_dmn_rec_var -C -H in.nc
time[0]=1 lat[0]=-90 lon[0]=0 three_dmn_rec_var[0]=1
% ncks --no_nm_prn -v three_dmn_rec_var -C -H in.nc
1 -90 0 1
% ncks -Q -v three_dmn_rec_var -C -H in.nc
three_dmn_rec_var[0]=1
% ncks -Q --no_nm_prn -v three_dmn_rec_var -C -H in.nc
1
% ncks -V -v three_dmn_rec_var -C -H in.nc
1
http://nco.sf.net/nco.html#var_val
E. ncks --cdl --hdn now prints extended file format (i.e., the format
of the file or server supplying the data)as _SOURCE_FORMAT:
% ncks --hdn --cdl ~/nco/data/cmip5.nc
...
:_Format = "netCDF-4" ;
:_SOURCE_FORMAT = "NC_FORMAT_HDF5" ;
...
http://nco.sf.net/nco.html#hdn
F. Hyperslabs specifications now accept -0 (negative zero) to indicate
the last element of a dimension. This is consistent with existing
(Python) syntax where negative integers indicate the number of
elements from the last (-1 is penultimate, -0 is ultimate):
ncks -d lon,1,-0 in.nc out.nc # Second through last longitude
http://nco.sf.net/nco.html#hyp
G. NCO now supports chunking and deflation features of HDF4 datasets
when compiled with netCDF library 4.3.3-rc1 or later.
Earlier versions of netCDF do not support these features for HDF4.
Thanks to Unidata for continuing to improve HDF4 interoperability.
http://nco.sf.net/nco.html#hdf4
BUG FIXES:
A. Fix treatment of packed variables with missing values in ncra.
Versions 4.3.7-4.4.5 of ncra could produce (wildly) inaccurate
statistics when promoting (e.g., to double- from single-precision)
variables that are packed and that have missing values.
The two workarounds are to remove the scaling or to perform the
statistics in single precision with the --flt option.
The solution is to upgrade to NCO 4.4.6.
Thanks to user John for reporting this.
http://nco.sf.net#bug_ncra_dbl_pck_mss
B. Fix promotion of packed variables subject to minimization and
maximization operations in ncwa. Versions 4.3.7-4.4.5 of ncwa
incorrectly handled packed variables during these operations.
The two workarounds are to remove the scaling or to perform the
statistics in single precision with the --flt option.
The solution is to upgrade to NCO 4.4.6.
Thanks to user Frederic Laliberte for reporting this.
http://nco.sf.net#bug_ncwa_pck_min_max
C. ncrename now fails (as intended) when required objects not present.
Since version 4.3.x, ncrename incorrectly ignored missing objects.
Now missing objects are ignored only when prefixed with a period.
A missing object (variable, attribute, dimension, group) not
prefixed with period now causes an error as intended.
Thanks to user John for reporting this.
http://nco.sf.net/nco.html#ncrename
D. NCO now checks correct function to enable DAP with configure.
NCO has been checking the wrong function to see if netCDF was
enabled with DAP support since about NCO 4.3.1 (May 2013).
Thanks to Parker Norton and Filipe Fernandes for reporting this.
E. Fix --file_format switch as synonym for --fl_fmt
Thanks to Dave Allured for reporting this.
KNOWN PROBLEMS DUE TO NCO:
This section of ANNOUNCE reports and reminds users of the
existence and severity of known, not yet fixed, problems.
These problems occur with NCO 4.4.6 built/tested with netCDF
4.3.3-rc2 (20141001) on top of HDF5 hdf5-1.8.13 with:
cd ~/nco;./configure --enable-netcdf4 # Configure mechanism -or-
cd ~/nco/bld;make dir;make allinone # Old Makefile mechanism
A. NOT YET FIXED (NCO problem)
Correctly read arrays of NC_STRING with embedded delimiters in
ncatted arguments
Demonstration:
ncatted -D 5 -O -a
new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc
~/foo.nc
ncks -m -C -v att_var ~/foo.nc
20130724: Verified problem still exists
TODO nco1102
Cause: NCO parsing of ncatted arguments is not sophisticated
enough to handle arrays of NC_STRINGS with embedded delimiters.
B. NOT YET FIXED (NCO problem?)
ncra/ncrcat (not ncks) hyperslabbing can fail on variables with
multiple record dimensions
Demonstration:
ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc
20140826: Verified problem still exists
20140619: Problem reported by rmla
Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
Workaround: Convert to fixed dimensions then hyperslab
KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:
A. NOT YET FIXED (netCDF4 or HDF5 problem?)
Specifying strided hyperslab on large netCDF4 datasets leads
to slowdown or failure with recent netCDF versions.
Demonstration with NCO <= 4.4.5:
time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
Demonstration with NCL:
time ncl < ~/nco/data/ncl.ncl
20140718: Problem reported by Parker Norton
20140826: Verified problem still exists
20140930: Finish NCO workaround for problem
Cause: Slow algorithm in nc_var_gets()?
Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
Workaround #2: Convert file to netCDF3 first, then use stride
B. NOT YET FIXED (would require DAP protocol change?)
Unable to retrieve contents of variables including period '.' in name
Periods are legal characters in netCDF variable names.
Metadata are returned successfully, data are not.
DAP non-transparency: Works locally, fails through DAP server.
Demonstration:
ncks -O -C -D 3 -v var_nm.dot -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to
find variable
20130724: Verified problem still exists.
Stopped testing because inclusion of var_nm.dot broke all test scripts.
NB: Hard to fix since DAP interprets '.' as structure delimiter in
HTTP query string.
Bug report filed: https://www.unidata.ucar.edu/jira/browse/NCF-47
C. NOT YET FIXED (would require DAP protocol change)
Correctly read scalar characters over DAP.
DAP non-transparency: Works locally, fails through DAP server.
Problem, IMHO, is with DAP definition/protocol
Demonstration:
ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc
20120801: Verified problem still exists
Bug report not filed
Cause: DAP translates scalar characters into 64-element (this
dimension is user-configurable, but still...), NUL-terminated
strings so MD5 agreement fails
"Sticky" reminders:
A. Pre-built Debian Sid & Ubuntu packages:
http://nco.sf.net#debian
B. Pre-built Fedora and CentOS RPMs:
http://nco.sf.net#rpm
C. Pre-built Mac binaries:
http://nco.sf.net#mac
D. Pre-built Windows (native) and Cygwin binaries:
http://nco.sf.net#windows
F. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g.,
HDF4: AMSR MERRA MODIS ...
HDF5: GLAS ICESat Mabel SBUV ...
HDF-EOS5: AURA HIRDLS OMI ...
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(
