The netCDF Operators NCO version 4.4.5 are ready.
http://nco.sf.net (Homepage)
This release is mainly a bugfix and stability release.
A few new features improve some corner-cases:
NCO now supports longer lists of input files, DAP files on HTTPS
servers, and the CF ancillary_variables convention.
Work on NCO 4.4.6 is underway, still focused on stability and speed.
This includes more netCDF4 mop-up and new chunking features.
Enjoy,
Charlie
NEW FEATURES (full details always in ChangeLog):
A. NCO now supports stdin input file lists up to ~50 MB long.
This applies to all NCO multi-file operators (ncra, ncea, ncrcat).
The previous limit was 10 MB. The new limit suffices for 30 years
of hourly datasets where each dataset name is 192 bytes long.
One NASA user is averaging 10 years of data, or 87600 files.
Who will be the first user to break the million file barrier?
http://nco.sf.net/nco.html#stdin
B. NCO now supports the CF ancillary_variables convention described at
http://cfconventions.org/1.6.html#ancillary-data
This convention allows ancillary variables to be associated with
one or more primary variables. NCO attaches any such variables to
the extraction list along with the primary variable and its
coordinates and bounds, if any.
http://nco.sf.net/nco.html#ncl
C. NCO now accesses DAP files on HTTPS servers. Previously NCO would
only test filenames beginning with "http://"; for DAP access.
Now NCO also tests filenames beginning with "https://"; for DAP.
If you use DAP servers with a protocol besides HTTP or HTTPS please
tell us so we can enable NCO for those services too.
http://nco.sf.net/nco.html#dap
D. ncks --grp_xtr_var_xcl: Group Extraction Variable Exclusion
This new switch helps control variable and group extraction.
Normally NCO works in "intersection mode" where the objects
extracted are the intersection of arguments to -g and -v.
And the --union switch can be used to make their union.
As of 4.4.4 (June, 2014), ncks (alone) supports an option to
include specified groups yet exclude specified variables.
The --grp_xtr_var_xcl switch (with long option equivalent --gxvx
extracts all contents of groups given as arguments to
-g except for variables given as arguments to -v.
Use this when one or a few variables in hierarchical files are not
to be extracted, and all other variables are. This is useful when
coercing netCDF4 files into netCDF3 files such as with converting,
flattening, or dismembering files.
ncks --grp_xtr_var_xcl -g g1 -v v1 # Extract all of group g1 except v1
http://nco.sf.net/nco.html#gxvx
E. New chunking policy r1d changes chunksize of 1-D record variables
(and no other variables) to specified (with --cnk_dmn) chunksize.
Any specified record dimension chunksizes will be applied to 1-D
record variables only. Other dimensions may be chunked with their
own --cnk_dmn options that will apply to all variables.
For example, ncks --cnk_plc=r1d --cnk_dmn=time,1000.
This sets time chunks to 1000 only in 1-D record variables.
Without the r1d policy, time chunks would change in all variables.
Not to be confused with chunking map rd1.
Thanks to James Adams for suggesting this.
http://nco.sf.net/nco.html#r1d
BUG FIXES:
A. ncrename: fix unsafe pointer de-reference and improve diagnostics
B. Fix problem introduced in 4.4.0 where sanitizing of filenames with
colons confused MS Windows interpretation of volume names, e.g.,
"C:\foo" became "C\:\foo" and caused write error. Workaround is to
use --no_tmp_fl or upgrade to 4.4.5. Thanks to James Adams for
reporting.
C. Always send correct dimension sizes to chunker when hyperslabbing.
Fixes a bug reported where chunking and hyperslabbing played
cruelly together. Thanks to Parker Norton for reporting.
http://nco.sf.net#bug_cnk_hyp
D. ncpdq: fix mishandling of dimension reversal arguments
Versions 4.3.2-4.4.4 of ncpdq could fail to correctly parse some
valid forms of the dimension reversal syntax. Failure resulted,
so no bad data was written. Solution is to upgrade to 4.4.5.
Thanks to Fan Fang for reporting.
KNOWN PROBLEMS DUE TO NCO:
This section of ANNOUNCE reports and reminds users of the
existence and severity of known, not yet fixed, problems.
These problems occur with NCO 4.4.5 built/tested with netCDF
4.3.3-rc1 on top of HDF5 hdf5-1.8.13 with these methods:
cd ~/nco;./configure --enable-netcdf4 # Configure mechanism -or-
cd ~/nco/bld;make dir;make allinone # Old Makefile mechanism
A. NOT YET FIXED (NCO problem)
Correctly read arrays of NC_STRING with embedded delimiters in
ncatted arguments
Demonstration:
ncatted -D 5 -O -a
new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc
~/foo.nc
ncks -m -C -v att_var ~/foo.nc
20130724: Verified problem still exists
TODO nco1102
Cause: NCO parsing of ncatted arguments is not sophisticated
enough to handle arrays of NC_STRINGS with embedded delimiters.
B. NOT YET FIXED (NCO problem?)
ncra/ncrcat (not ncks) hyperslabbing can fail on variables with
multiple record dimensions
Demonstration:
ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc
20140826: Verified problem still exists
20140619: Problem reported by rmla
Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
Workaround: Convert to fixed dimensions then hyperslab
C. NOT YET FIXED (NCO problem?)
Correctly handle stride argument on netCDF4 datasets
Specifying strided hyperslab on netCDF4 datasets leads
to enormous slowdown or failure with recent NCO versions.
Demonstration:
ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
20140826: Verified problem still exists
20140718: Problem reported by Parker Norton
Cause: Unclear. Interaction of stride and MSA?
Workaround: Convert file to netCDF3 first, then use stride
KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:
A. NOT YET FIXED (netCDF library problem)
Probe hidden attributes (chunking, compression) of HDF4 files
Demonstration:
ncdump -h -s ~/nco/data/hdf.hdf # (dies)
ncks -m ~/nco/data/hdf.hdf # (works by avoiding fatal calls)
20131230: Verified problem still exists
Cause: some libnetCDF library functions fail on HDF4 file inquiries.
Bug report filed: netCDF #HZY-708311 ncdump/netCDF4 segfaults
probing HDF4 file
Tracking tickets NCF-272, NCF-273
B. NOT YET FIXED (would require DAP protocol change?)
Unable to retrieve contents of variables including period '.' in name
Periods are legal characters in netCDF variable names.
Metadata are returned successfully, data are not.
DAP non-transparency: Works locally, fails through DAP server.
Demonstration:
ncks -O -C -D 3 -v var_nm.dot -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to
find variable
20130724: Verified problem still exists.
Stopped testing because inclusion of var_nm.dot broke all test scripts.
NB: Hard to fix since DAP interprets '.' as structure delimiter in
HTTP query string.
Bug report filed: https://www.unidata.ucar.edu/jira/browse/NCF-47
C. NOT YET FIXED (would require DAP protocol change)
Correctly read scalar characters over DAP.
DAP non-transparency: Works locally, fails through DAP server.
Problem, IMHO, is with DAP definition/protocol
Demonstration:
ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc
20120801: Verified problem still exists
Bug report not filed
Cause: DAP translates scalar characters into 64-element (this
dimension is user-configurable, but still...), NUL-terminated
strings so MD5 agreement fails
"Sticky" reminders:
A. Pre-built Debian Sid & Ubuntu packages:
http://nco.sf.net#debian
B. Pre-built Fedora and CentOS RPMs:
http://nco.sf.net#rpm
C. Pre-built Mac binaries:
http://nco.sf.net#mac
D. Pre-built Windows (native) and Cygwin binaries:
http://nco.sf.net#windows
F. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g.,
HDF4: AMSR MERRA MODIS ...
HDF5: GLAS ICESat Mabel SBUV ...
HDF-EOS5: AURA HIRDLS OMI ...
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(
