« Unidata is looking... | Main | Unidata Summer Stude... »

05 December 2019

Version 4.9.0 of the netCDF Operators (NCO) has been released. NCO is an Open Source package that consists of a dozen standalone, command-line programs that take netCDF files as input, then operate (e.g., derive new data, average, print, hyperslab, manipulate metadata) and output the results to screen or files in text, binary, or netCDF formats.

The NCO project is coordinated by Professor Charlie Zender of the Department of Earth System Science, University of California, Irvine. More information about the project, along with binary and source downloads, are available on the SourceForge project page .

From the release message:

Version 4.9.0 includes new regridder features, JSON, and ncap2 fixes. Notably, this version simplifies the weight-application interface, parallelizes ncremap in stand-alone mode, and completely revamps, improves, and accelerates the sub-gridscale weight application algorithm.

The NCO conservative weight-generator algorithm also debuts. Previously, ncremap always used ESMF or TempestRemap to make weights. Use 'ncremap -a nco' to get NCO's conservative weight generation. NCO will always support ESMF and TR which are more time-tested. However, NCO has some unique features, like an option to output the overlap mesh in SCRIP format, and it performs best for some grids.

New Features

ncremap has been refactored for full parallelism. Previously the standalone ncremap could only run parallelized when invoked with pure weight application requests. Special processing like CLM/ELM sub-gridscale regridding or MPAS juggling only worked with single file invocations (which is what ncclimo always requests). Now the standalone ncremap is parallelized and adheres to --job_nbr control for all regridding in standalone invocations, meaning that regridding lists of files offline with MPI or background parallelism is now much quicker:
```
ls cam/*.nc | ncremap --job_nbr=4 --par_typ=mpi -m map.nc
ls mpas/*.nc | ncremap -P mpas -j 4 -p mpi -m map.nc
ls elm/*.nc | ncremap --sgs_frc=landfrac -j 16 -m map.nc
```
http://nco.sf.net/nco.html#ncremap
ncremap has a new API for sub-gridscale (SGS) regridding. Previously ncremap employed a convoluted and slow procedure to remap data with sub-gridscale fractions, e.g., CLM/ELM, CICE, and MPAS-Seaice data. That procedure and ncremap API used to require specification of the source and destination grids. The new, faster ncremap SGS procedure requires only the map-file and the name of the sub-gridscale fraction field. The -P sgs option is no longer required, thought is still accepted for back-compatibility.
```
ncremap --sgs_frc=landfrac -m map.nc # CLM/ELM
ncremap --sgs_frc=frc.nc/landfrac -m map.nc # CLM/ELM external
ncremap -P elm -m map.nc # CLM/ELM alternative
ncremap --sgs_frc=aice --sgs_msk=tmask -m map.nc # CICE
ncremap -P cice -m map.nc # CICE alternative
ncremap --sgs_frc=timeMonthly_avg_iceAreaCell -m map.nc # MPAS-Seaice
ncremap -P mpasseaice -m map.nc # MPAS-Seaice alternative
```
http://nco.sf.net/nco.html#ncremap
ncremap supports the new NCO weight generation algorithm. The NCO algorithm is first-order conservative and compares well in accuracy with other FV algorithms such as ERWG and TR. Special thanks to Henry Butowsky for his dedication to this. We evaluated ERWG "conserve", NCO "nco", and TR "mono" with --chk_map (see below) on the E3SMv1 lo-res and hi-res grids. The three algorithms generally agree to 10-13 significant digits. ESMF won for atm->atm remapping, and NCO for ocn->atm remapping, both by razor-thin margins. YMMV. We hope users will try the NCO algorithm and send feedback. NCO weight-generation is threaded and scales well to 3-4 threads.
```
ncremap -a nco -s grd_src.nc -d grd_dst.nc -m map.nc in.nc out.nc
ncremap -t 4 -a nco -s grd_src.nc -d grd_dst.nc -m map.nc
```
http://nco.sf.net/nco.html#ncremap
ncks supports a new option --chk_map to evaluate the quality of regridding weights. This option works with map-files (not grid-files) in ESMF/CMIP6-compliant format (i.e., sparse matrix S and coordinates [xy][ab]_[cv]). When invoked with the additional --area_wgt option (also new), the evaluation statistics are area-weighted and thus reflect exactly the global-mean/min/max/ mebs/rms/sdn biases expected when regridding globally uniform fields. This tool makes it easier to objectively assess weight-generation algorithms, and will hopefully assist in their improvement.
```
ncks --chk_map map.nc
ncks --chk_map --area_wgt map.nc
```
http://nco.sf.net/nco.html#chk_map http://nco.sf.net/nco.html#area_wgt
ncremap works with newer MPAS Ocean files that contain BGC dimensions like R3 and FOUR.
```
ncremap -P mpas -m map_bilin.nc in.nc out.nc
```
http://nco.sf.net/nco.html#ncremap
ncremap now propagates all netCDF formats except CDF5 to TempestRemap (TR) (which does not yet accept or produce CDF5 files AFAICT).
http://nco.sf.net/nco.html#tr
ncremap omits the TR --volumetric flag from fv2se_stt maps. TempestRemap author Paul Ullrich recommends this to keep fv2se_stt maps consistent (producing output in same range) with fv2se_flx maps.
```
ncremap -a fv2se_stt -s grd_src.nc -d grd_dst.nc -m map.nc in.nc out.nc
```
http://nco.sf.net/nco.html#ncremap
ncremap allows simultaneous invocation of horizontal and vertical regridding. Yes, now ncremap can automagically regrid any file to any new horizontal and vertical grid at the same time. Simply supply both a map-file and a vertical grid-file:
```
ncremap -m map.nc --vrt=vrt_grd.nc in.nc out.nc
```
http://nco.sf.net/nco.html#vrt
http://nco.sf.net/nco.html#ncremap
ncremap/ncclimo have deprecated support for options named for the short-lived ALM land surface model. Use 'elm' instead of 'alm',
```
ncremap -P elm -m map.nc
```
http://nco.sf.net/nco.html#ncremap
NCO supports flexible options to specify non-default names for vertical grid coordinates in input and output (vertically interpolated) data files.
```
ncks --rgr plev_nm=vrt_nm --vrt=vrt_grd.nc in.nc out.nc
ncremap -n '--rgr plev_nm=vrt_nm' --vrt=vrt_grd.nc in.nc out.nc
```
http://nco.sf.net/nco.html#lev_nm
ncks supports a new option --chk_nan to quickly find the location of any NaN values in a dataset. Thanks to Matthew Thompson of NASA for this suggestion.
```
ncks --chk_nan in.nc
```
http://nco.sf.net/nco.html#chk_nan