The netCDF operators NCO version 2.9.9 are ready.
http://nco.sf.net (Homepage)
http://dust.ess.uci.edu/nco (Homepage "mirror")
First, the NCO->SDO proposal was funded by NSF.
A new section on the NCO homepage describes the project and contains
pointers to the proposal and two job ads. For more details, see
http://nco.sf.net#prp_sei
There are two significant feature improvements in version 2.9.9:
1. ncbo now supports threading.
Manually configure threading with -t (as with ncra, ncpdq, ncea), or
ncbo will pick a default for you.
2. ncpdq has powerful packing and unpacking capabilities
ncpdq stands for "Permute Dimensions Quickly" and now "Pack Data
Quietly" too. See http://nco.sf.net/nco.html#ncpdq for full info.
Prior to this release, ncap sported the only packing features in NCO.
However, ncap only packs one variable at a time.
ncpdq is more consistent with the NCO file-at-a-time paradigm.
To pack the entire file in.nc into output file out.nc use
ncpdq in.nc out.nc
ncpdq -P pck_plc -M pck_map in.nc out.nc
The main packing options to ncpdq are -P pck_plc and -M pck_map.
pck_plc is the "packing policy". Currently implemented policies are
all_new: Pack unpacked variables, re-pack packed variables (default)
all_xst: Pack unpacked variables, copy packed variables
xst_new: Re-pack packed variables, copy unpacked variables
upk: Unpack packed variables, copy unpacked variables
pck_map is the "packing map" which determines how variables of each
numeric type are packed. Currently implemented maps are
flt_sht: Pack floating precision types to NC_SHORT, pack nothing else (Default)
[NC_DOUBLE,NC_FLOAT]->NC_SHORT,
[NC_INT,NC_SHORT,NC_CHAR,NC_BYTE]->unaltered
flt_byt: Pack floating precision types to NC_BYTE: pack nothing else
[NC_DOUBLE,NC_FLOAT]->NC_BYTE,
[NC_INT,NC_SHORT,NC_CHAR,NC_BYTE]->unaltered
hgh_sht: Pack higher precision types to NC_SHORT, pack nothing else
[NC_DOUBLE,NC_FLOAT,NC_INT]->NC_SHORT,
[NC_SHORT,NC_CHAR,NC_BYTE]->unaltered
hgh_byt: Pack higher precision types to NC_BYTE, pack nothing else
[NC_DOUBLE,NC_FLOAT,NC_INT,NC_SHORT]->NC_BYTE,
[NC_CHAR,NC_BYTE]->unaltered
nxt_lsr: Convert each type to next lesser precision
NC_DOUBLE->NC_INT, [NC_FLOAT,NC_INT]->NC_SHORT,
[NC_CHAR,NC_BYTE]->unaltered
The default "all_new" packing policy with the default "flt_sht"
packing map reduces the typical NC_FLOAT-dominated file size by 50%.
"flt_byt" packing reduces an NC_DOUBLE-dominated file by 87%.
One nice feature of (lossy) scale_factor/add_offset packing is that
additional, loss-less packing algorithms perform well on top of it.
Caveats:
1. The interaction of packing and missing_values is complex.
Test the missing_value behavior by performing a pack/unpack cycle
to make sure your data that are missing _stay_ missing and data
that are not misssing do not join the Alabama Air National Guard.
This may lead you to elect a new missing_value.
2. Packing into NC_CHAR (with, e.g., flt_chr) is actually allowed
but is not recommended and so is not documented/advertised.
Pack into NC_BYTE (with, e.g., flt_byt) instead.
FYI, the plan is to stabilize recent changes in NCO 3.0.0.
Then we will start development of the grid-enabled features of SDO.
Enjoy,
Charlie
Feature changes:
1. ncbo supports threading
1. ncpdq packs/unpacks
Bug-fixes:
1. Fix regex capability on Mac OS X
Build changes:
1. None
Other user-visible changes:
1. Documentated thread options http://nco.sf.net/nco.html#openmp
2. Yummier ncpdq documentation http://nco.sf.net/nco.html#ncpdq
User-invisible changes:
1. None
--
Charlie Zender, surname@xxxxxxx, (949) 824-2987, Department of Earth
System Science, University of California, Irvine CA 92697-3100
