« Project Manager:... | Main | Unidata McIDAS v2016... »

13 May 2016

Version 4.6.0 of the netCDF Operators (NCO) has been released. NCO is an Open Source package that consists of a dozen standalone, command-line programs that take netCDF files as input, then operate (e.g., derive new data, average, print, hyperslab, manipulate metadata) and output the results to screen or files in text, binary, or netCDF formats.

The NCO project is coordinated by Professor Charlie Zender of the Department of Earth System Science, University of California, Irvine. More information about the project, along with binary and source downloads, are available on the SourceForge project page.

From the release message:

4.6.0 may be most notable for the debut of ncclimo , a new operator that generates climatologies from monthly-mean input. Perhaps it's a tie with ncap2 , which has a singularly useful new feature: variable lists/pointers. ncap2 also has a reduced memory footprint and a function to simplify adding CF-bounds variables (Thanks Henry!). As usual, ncremap continues to accrue useful features, the most notable of which is learning grid information from the CF "coordinates" attribute, if any.

New Features

ncclimo produces climatological monthly means, seasonal means, annual mean, and optionally regrids all these files. Like ncremap, ncclimo is actually a front-end script to manage the complexity of invoking the underlying operators (ncks for ncremap and ncra for ncclimo ). ncclimo produces climatologies starting from monthly-mean files of most CESM-like model and observational data formats.
Invoke ncclimo with, e.g.,
```
ncclimo -s start_yr -e end_yr -c run_id -i drc_in -o drc_out
ncclimo -s 5 -e 10 -c hist -m ocn -i drc_in -o drc_out
ncclimo -s 1980 -e 2016 -c merra2_198001.nc -i drc_in -o drc_out
```
ncclimo has sophisticated parallelism and regridding options. Datasets tested include ALM, CAM, CLM, CICE, CISM, CLM, MPAS-I, MPAS-O, and POP. ncclimo resulted from work with the DOE ACME project, and we are grateful for their support.
http://nco.sf.net/nco.html#ncclimo
http://nco.sf.net/nco.html#merra2
ncap2 now works with pointers to variables and attributes. This allows, e.g., loops over arbitrary sets of variables, and make creating aggregations of variables much simpler. An ncap2 script that converts all float variables to double precision:
```
@all=get_vars_in();
*sz=@all.size();
for(idx=0;idx<sz;idx++){
  @var_nm=sprint(@all(idx));
  if(*@var_nm.type() == NC_FLOAT) *@var_nm=*@var_nm.double();
}
```
Congratulations Henry for finishing this impressive functionality! More documentation will be added in the coming weeks. For now,
http://nco.sf.net/nco.html#ncap2
ncap2 has reduced memory usage. Scalars are no longer stretched to conform to arrays prior to arithmetic.
ncap2 has a new function to create CF-compliant bounds:
```
defdim("bnd_dmn",2);
bounds_var=make_bounds(crd_var,$bnd_dmn,"bounds_var_nm");
```
The "bounds_var_nm" is optional. Passing it causes make_bounds() to add the attribute crd_var@bounds="bounds_var_nm".
http://nco.sf.net/nco.html#make_bounds
ncap2 now propagates metadata when Left-Hand-Casting variables. Previous versions did not propagate metadata to LHC variables. This fix makes converting the _type_ of a variable easier. For example, to change one_dmn_rec_var to NC_SHORT, use
```
ncap2 -s 'one_dmn_rec_var[time]=short(one_dmn_rec_var)' in.nc out.nc
```
The output file will contain the original metadata.
http://nco.sf.net/nco.html#ncap2
ncatted new mode "nappend" appends only to existing attributes. If the attribute does not yet exist, it is not created. Same as "append" mode, except "append" will create attributes.
```
ncatted -a long_name,T,n,c,' nappended text' in.nc
```
http://nco.sf.net/nco.html#ncatted
ncpdq now has a third unpacking mode. Access non-default modes with --upk=[0,1,2] where Mode 0: Applies netCDF convention. Default mode:
upk=scale_factor*pck+add_offset
Mode 1: HDF4 MODIS MOD08 convention:
upk=scale_factor*(pck-add_offset)
Mode 2: HDF4 MODIS MOD13 convention:
upk=(pck-add_offset)/scale_factor
Yes, be careful unpacking HDF4 data!
http://nco.sf.net/nco.html#hdf_upk
ncra can now honor the CF convention for climatology bounds. Currently this is opt-in with the --cb or --c2b switches. --cb causes ncra to:
1. Add a "climatology" attribute with value "climatology_bounds" to the time coordinate, if necessary
2. Remove the "bounds" from the time coordinate, if necessary
3. Output a variable named "climatology_bounds" with values that are minima/maxima of the input time coordinate bounds
4. Omit any input time coordinate bounds attribute and variable
5. Ensure cell_methods attribute for all variables is appropriate --c2b is like --cb except --c2b converts the input "climatology" bounds to a non-climatology "bounds" in the output.
Use --c2b when averaging sub-sampled climatologies to produce a continuous (non-climatologically sub-sampled) mean.
```
ncra --cb *_01.nc clm_JAN.nc
ncra --cb clm_DEC.nc clm_JAN.nc clm_FEB.nc clm_DJF.nc
ncra --c2b clm_DJF.nc clm_MAM.nc clm_JJA.nc clm_SON.nc clm_ANN.nc
```
Depending on feedback, we may make --cb the default
http://nco.sf.net/nco.html#cb
ncremap now defaults to $TMPDIR for storing intermediate files. This is user-configurable with the -U switch.
http://nco.sf.net/nco.html#ncremap
ncks can now ingest and de-interleave ENVI images in BIL, BSQ, and BIP formats and store them as any netCDF type. ncks is a viable (and faster) replacement for that small subset of tasks normally done by the venerable GDAL toolkit (gdal_translate in particular). Thanks to David LeBauer of the University of Illinois/NCSA and DOE TERRAREF for supporting this feature.
```
ncks --trr_wxy=926,1600,1 --trr typ_in=NC_USHORT --trr ntl_in=bil \
 --trr_in=in.nc foo.nc out.nc
```
http://nco.sf.net/nco.html#terraref
NCO now treats as associated coordinates all variables listed in "grid_mapping" attributes. Variables such as "char albers_conical_equal_area" are automatically extracted along with variables that them. ncecat will not wrap mappings in a record dimension.
http://nco.sf.net/nco.html#grid_mapping
ncremap and ncks now follow the CF "coordinates" convention when inferring grids and remapping. One specifies the variable that may have the "coordinates" attribute, and NCO will inspect and apply the "coordinates" convention for that variable for grid generation and remapping.
```
ncremap -V var_nm -s src.nc -d dst.nc -m map.nc
ncremap -V var_nm --map map.nc in.nc out.nc
ncks --rgr_var=var_nm --rgr nfr=y --rgr grid=grd.nc in.nc foo.nc
ncks --rgr_var=var_nm --map map.nc in.nc out.nc
```
http://nco.sf.net/nco.html#ncremap
http://nco.sf.net/nco.html#regrid