The netCDF Operators NCO version 4.3.9 are ready.
http://nco.sf.net (Homepage)
http://dust.ess.uci.edu/nco (Homepage "mirror")
We are pleased to report the birth of the operator nces.
nces is for netCDF Ensemble Statistics.
nces includes the existing functionality of ncea and extends it by
allowing groups to be treated as an unlimited dimension of datasets.
nces is really ncea with a powerful new feature and a new name.
I will debut nces at AGU (Friday 12/13 11:35 AM Moscone West 2010).
See below for more info on the feature. Regarding the name...
We renamed ncea as nces to more accurately describe the operator.
For historical reasons, NCO is full of misnamed operators:
NCO itself is misnamed because the operators now work well on HDF (4
and 5) files, not just netCDF files. We tried to change their name
to SDO (Scientific Data Operators) but the new name didn't stick.
And the "averagers" (ncea, ncra, ncwa) are all misnamed because they
all do many statistics (min/max/ttl/rms...) besides averages.
This feature addition is an opportune time to rename ncea as nces.
ncea is now deprecated, though will remain supported for some time.
We faced the same problem of back-compatibility when we renamed ncdiff
to ncbo many years ago. We will use the same method. Invoking 'nces'
produces the same results as the old 'ncea' so scripts do not break.
Under the hood, 'ncea' invokes 'nces --nsm_fl', just as 'ncdiff'
invokes 'ncbo --op_typ=subtract'. To cause nces to utilize the new
group ensemble mode, invoke it with 'nces --nsm_grp'. This is similar
to how ncecat has two modes, record aggregation (the default) and
group aggregation, invoked with 'ncecat --rag' and 'ncecat --gag',
respectively.
This version also improves ncks support for XML (NcML) output.
All output now validates without errors with the NcML 2.2 schema.
Happy NcML'ing! There is also a bugfix for NaNs in ncatted.
Work on NCO 4.4.0 is underway. It focuses on stability and speed.
There will be some netCDF4 mop-up, including netCDF4 support for
various options (all -X, ncwa -w and -m), and, possibly, improved
support for HDF4 files. Heads-up: Unless users report unforeseen
difficulties transitioning from ncea to nces, NCO 4.4.0 will rename
ncra as ncrs and ncwa as ncws (Record Statistics and Weighted
Statistics, respectively).
Enjoy,
Charlie
"New stuff" in 4.3.9 summary (full details always in ChangeLog):
NEW FEATURES:
A. ncea is now nces or nces --nsm_fl, meaning ensembles of files.
Use nces --nsm_grp, for ensembles of groups, a new feature.
The same input is valid in both modes. Members of a group ensemble
are groups that share the same structure, parent group, and nesting
level. Their contents usually have different values because they
are realizations of replicated experiments.
In group-ensemble mode nces computes the statistics across
the ensemble of groups, which may span multiple input files.
Files may contain members of multiple ensembles. However, all
ensembles must have at least one member in the first input file.
Group ensembles behave as a pseudo-record dimension, in that a
group ensemble may comprise an arbitrary and extensible number of
realizations in each file, and may be composed of multiple files.
Output statistics are stored in the parent group by default.
If the ensemble members are /cesm/01 and /cesm/02, then the
computed statistic will be in /cesm in the output file.
The --nsm_sfx option instructs nces to instead store output in
a new child group of the parent created by attaching the suffix
to the parent group's name, i.e., /cesm/cesm_avg:
nces mdl1.nc mdl2.nc mdl3.nc out.nc # Same as ncea
nces --nsm_fl mdl1.nc mdl2.nc mdl3.nc out.nc # Same as ncea
nces --nsm_grp mdl1.nc mdl2.nc mdl3.nc out.nc
nces --nsm_grp --nsm_sfx='_avg' mdl1.nc mdl2.nc mdl3.nc out.nc
http://nco.sf.net/nco.html#nces
B. Much improved XML/NcML standards compliance and flexibility.
Separators for XML now user-configurable with --xml_spr_chr and
--xml_spr_nmr for character types (i.e., strings) and numeric
types (i.e., numbers), respectively:
ncks --xml_spr_chr='***' in.nc # Override default separator '*'
ncks --xml_spr_nmr=', ' in.nc # Override default separator ' '
ncks dumps all netCDF3/netCDF4 groups and atomic types
(as usual with NCO, netCDF4 non-atomic types are ignored).
Output validates without errors against NcML 2.2 schema.
Java/NcML users please test and and give us feedback!
http://nco.sf.net/nco.html#xml
BUG FIXES:
A. Fix ncatted so it does not treat zero as NaN when changing a
missing value (_FillValue) from NaN to something else.
Problem reported by Jianfu Pan.
B. Correct XML-mode issues, many reported by Aleksandar Jelenak
1. Print </variable> in metadata mode
2. Adds type="typ_nm" attribute to non-string attribute elements
3. Add separator="foo" element when separator is not whitespace
4. Fix int64-type printing
5. Use NcML not CDL type names
6. Add hidden _Unsigned attribute for unsigned types
7. Add hidden _FillValue attribute for unsigned types
8. Do not print underscores for missing data in XML mode
9. Add shape="" to scalar variables (cough, cough)
KNOWN ISSUES NOT YET FIXED:
This section of ANNOUNCE reports and reminds users of the
existence and severity of known, not yet fixed, problems.
These problems occur with NCO 4.3.7 built/tested with netCDF
4.3.1-rc4 snapshot 20131106 on top of HDF5 hdf5-1.8.9 with these
methods:
cd ~/nco;./configure --enable-netcdf4 # Configure mechanism -or-
cd ~/nco/bld;make dir;make all;make ncap2 # Old Makefile mechanism
A. NOT YET FIXED
netCDF4 library fails when renaming dimension and variable using
that dimension, in either order. Works fine with netCDF3.
Hence coordinate renaming does not work with netCDF4 files.
Problem with netCDF4 library implementation.
Demonstration:
ncks -O -4 -v lat_T42 ~/nco/data/in.nc ~/foo.nc
ncrename -O -D 2 -d lat_T42,lat -v lat_T42,lat ~/foo.nc ~/foo2.nc #
Breaks with "NetCDF: HDF error"
ncks -m ~/foo.nc
20130724: Verified problem still exists
Bug report filed: netCDF #YQN-334036: problem renaming dimension and
coordinate in netCDF4 file
Workaround: Use ncrename twice; first rename the variable, then
rename the dimension.
B. NOT YET FIXED (would require DAP protocol change?)
Unable to retrieve contents of variables including period '.' in name
Periods are legal characters in netCDF variable names.
Metadata are returned successfully, data are not.
DAP non-transparency: Works locally, fails through DAP server.
Demonstration:
ncks -O -C -D 3 -v var_nm.dot -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to
find variable
20130724: Verified problem still exists.
Stopped testing because inclusion of var_nm.dot broke all test scripts.
NB: Hard to fix since DAP interprets '.' as structure delimiter in
HTTP query string.
Bug report filed: https://www.unidata.ucar.edu/jira/browse/NCF-47
C. NOT YET FIXED (would require DAP protocol change)
Correctly read scalar characters over DAP.
DAP non-transparency: Works locally, fails through DAP server.
Problem, IMHO, is with DAP definition/protocol
Demonstration:
ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p
http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc
20120801: Verified problem still exists
Bug report not filed
Cause: DAP translates scalar characters into 64-element (this
dimension is user-configurable, but still...), NUL-terminated
strings so MD5 agreement fails
D. NOT YET FIXED (NCO problem)
Correctly read arrays of NC_STRING with embedded delimiters in
ncatted arguments
Demonstration:
ncatted -D 5 -O -a
new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc
~/foo.nc
20130724: Verified problem still exists
TODO nco1102
Cause: NCO parsing of ncatted arguments is not yet sophisticated
enough to handle arrays of NC_STRINGS with embedded delimiters.
E. NOT YET FIXED
Report correct chunking and compression information for HDF4 files
Demonstration:
ncdump -h -s MOP01-20121231-L1V3.34.10.hdf
ncks -m MOP01-20121231-L1V3.34.10.hdf
20131007: Verified problem still exists
Cause: some libnetCDF library functions fail on HDF4 file inquiries.
Bug report filed: netCDF #HZY-708311 ncdump/netCDF4 segfaults probing
HDF4 file
Tracking tickets NCF-272, NCF-273
"Sticky" reminders:
A. Pre-built, up-to-date Debian Sid & Ubuntu packages:
http://nco.sf.net#debian
B. Pre-built Fedora and CentOS RPMs:
http://nco.sf.net#rpm
C. Pre-built Windows (native) and Cygwin binaries:
http://nco.sf.net#windows
D. Pre-built AIX binaries:
http://nco.sf.net#aix
E. Did you try SWAMP (Script Workflow Analysis for MultiProcessing)?
SWAMP efficiently schedules/executes NCO scripts on remote servers:
http://swamp.googlecode.com
SWAMP can work command-line operator analysis scripts besides NCO.
If you must transfer lots of data from a server to your client
before you analyze it, then SWAMP will likely speed things up.
F. NCO support for netCDF4 features is tracked at
http://nco.sf.net/nco.html#nco4
NCO supports netCDF4 atomic data types, compression, chunking, and
groups.
G. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g.,
NASA AURA HIRDLS HDF-EOS5
NASA ICESat GLAS HDF5
NASA MERRA HDF4
NASA MODIS HDF4
NASA SBUV HDF5...
--
Charlie Zender, Earth System Sci. & Computer Sci.
University of California, Irvine 949-891-2429 )'(
