« NSF Unidata Update:... | Main | NetCDF operators... »

02 July 2024

Version 5.2.6 of the netCDF Operators (NCO) has been released. NCO is an Open Source package that consists of a dozen standalone, command-line programs that take netCDF files as input, then operate (e.g., derive new data, average, print, hyperslab, manipulate metadata) and output the results to screen or files in text, binary, or netCDF formats.

The NCO project is coordinated by Professor Charlie Zender of the Department of Earth System Science, University of California, Irvine. More information about the project, along with binary and source downloads, are available on the SourceForge project page.

From the release message:

Version 5.2.6 contains a one-line bugfix to ncremap in 5.2.5. Unfortunately that horse had already escaped the barn. There is nothing else new in 5.2.6, so the release notes below repeat those for 5.2.5 so people can get a quick summary of new features since the previous widely distributed version, 5.2.4.

New Features

All numeric operators now support --qnt_alg=alg_nm to request an NCO quantization algorithm by name. Previously one had to use the --baa option with an obscure undocumented integer code for each algorithm. Now quantization algorithms can be requested by the English names. alg_nm can be BitGroom, Granular BitRound, BitShave, BitSet, DigitRound, BitGroomRound, HalfShave, BruteForce, BitRound or common synonyms for these names, e.g., set, shave, "bit round", btg, etc. The names are case-insensitive:
```
ncks -7 -L 1 --qnt default=3 in.nc out.nc # Granular BitRound (NSD)
ncks -7 -L 1 --qnt_alg=btg --qnt default=3 in.nc out.nc # BitGroom (NSD)
ncks -7 -L 1 --qnt_alg=shv --qnt default=3 in.nc out.nc # BitShave (NSD)
ncks -7 -L 1 --qnt_alg=set --qnt default=3 in.nc out.nc # BitSet (NSD)
ncks -7 -L 1 --qnt_alg=dgr --qnt default=3 in.nc out.nc # DigitRound (NSD)
ncks -7 -L 1 --qnt_alg=gbr --qnt default=3 in.nc out.nc # Granular BitRound (NSD)
ncks -7 -L 1 --qnt_alg=bgr --qnt default=3 in.nc out.nc # BitGroomRound (NSD)
ncks -7 -L 1 --qnt_alg=sh2 --qnt default=9 in.nc out.nc # HalfShave (NSB)
ncks -7 -L 1 --qnt_alg=brt --qnt default=3 in.nc out.nc # BruteForce (NSD)
ncks -7 -L 1 --qnt_alg=btr --qnt default=9 in.nc out.nc # BitRound (NSB)
```
This menagerie arose from research efforts. We recommend that others choose between BitRound, DigitRound, and Granular BitRound for real-world workflows. The others are of mainly research or historical interest only.
http://nco.sf.net/nco.html#qnt_alg
http://nco.sf.net/nco.html#qnt

CF-compliant metadata for quantization now includes maximum relative error (MRE) for the BitRound algorithm, for which MRE is 0.5*2^-NSB = 2^-(NSB+1). The MRE appears as the attribute "lossy_compression_maximum_relative_error" in each field's metadata:

ncks -7 -v ps,ts --qnt_alg=btr --qnt default=9 --qnt ps=13 --cmp='shf|zst' in.nc out.nc
ncks -m -C -v ps,ts,compression_info out.nc
netcdf out {
...
float ps(time,lat,lon) ;
  ps:standard_name = "surface_air_pressure" ;
  ps:units = "Pa" ;
  ps:lossy_compression = "compression_info" ;
  ps:lossy_compression_nsb = 13 ;
  ps:lossy_compression_maximum_relative_error = 6.103516e-05f ;

float ts(time) ;
  ts:standard_name = "surface_temperature" ;
  ts:units = "K" ;
  ts:lossy_compression = "compression_info" ;
  ts:lossy_compression_nsb = 9 ;
  ts:lossy_compression_maximum_relative_error = 0.0009765625f ;
} // group /

http://nco.sf.net/nco.html#qnt_alg
http://nco.sf.net/nco.html#qnt

ncremap and ncclimo now handle MPAS-A (Atmosphere) datasets. These operators have always supported MPAS Ocean, Sea-ice, and Land-Ice (all used by E3SM) datasets. This completes MPAS support. Use -P mpasa to indicate that datasets follow MPAS-A conventions. This allow ncremap to automatically permute the spatial dimensions into the correct order for regridding, and to differentiate itself from other MPAS datasets in terms of missing value treatment:
```
ncremap -P mpasa  --map=map.nc mpa.nc foo.nc # MPAS-A
ncremap -P mali   --map=map.nc mpli.nc foo.nc # MPAS-LI
ncremap -P mpaso  --map=map.nc mpo.nc foo.nc # MPAS-O
ncremap -P mpassi --map=map.nc mpsi.nc foo.nc # MPAS-SI
```
NB: Omitting the -P mpasX option on MPAS datasets works if the user explicitly permutes the horizontal dimensions, e.g.:
```
ncremap --pdq=Time,nVertLevels,nIsoLevelsT,nIsoLevelsZ,nCells \
--map=map.nc mpa.nc foo.nc # MPAS-A
```
Analogous workarounds apply to the other MPAS componenents when omitting the -P option, and -P mpas also works for generic MPAS. However, using -P mpasX results in the best output.
http://nco.sf.net/nco.html#MPAS
http://nco.sf.net/nco.html#ncremap
http://nco.sf.net/nco.html#pdq_opt
ncclimo now supports excluding the specified variable list (with -x or --xcl_var or --exclude) in timeseries mode. Previously this option only worked in climo mode. However, in timeseries mode this option requires invoking ncclimo with Bash version 4.0 or higher. NB: This works well across modern Linux machines, though MacOS still ships Bash 3.2.57 (from 2007!). MacOS users must put an updated Bash on their PATH before /bin/bash to access this features (all other features continue to work fine with older versions of Bash).
```
ncclimo --split --exclude -v FSNT,AODVIS,TREFHT \
-c v2.LR.historical_0101 -s 2013 -e 2014 \
-i ${DATA}/ne30/raw -o ${DATA}/ne30/clm
```
http://nco.sf.net/nco.html#xcl_var
ncremap now automatically tests whether vertical grid-file has a level(level) coordinate à la ERA5. If so, it treats this as a pure pressure coordinate. This is analogous to the treatment of the plev(plev) coordinate for NCEP files.
```
ncremap --vrt_out=vrt_prs_era5_L37.nc in.nc out.nc
```
http://nco.sf.net/nco.html#vrt
ncclimo updated its MPAS dataset filename construction option. Previously it constructed MPAS monthly datasets names like this:
```
${mdl_nm}.hist.am.timeSeriesStatsMonthly.${YYYY}-${MM}-01.nc
```
where mdl_nm is the canonical MPAS component name, e.g., mpaso. This yielded names consistent with MPAS v1 output like
```
"mpaso.hist.am.timeSeriesStatsMonthly.0001-02-01.nc"
```
and
```
"mpascice.hist.am.timeSeriesStatsMonthly.0001-02-01.nc"
```
Now ncclimo prepends the ${caseid}, if present, to the filename. This yields names consistent with E3SM v2 and v3 output like
```
"v2.LR.historical_0101.mpaso.hist.am.timeSeriesStatsMonthly.0001-02-01.nc"
```
and
```
"v2.LR.historical_0101.mpassi.hist.am.timeSeriesStatsMonthly.0001-02-01.nc"
```
To read MPAS filenames with different patterns, simply pipe the filenames to ncclimo:
```
ls *mpas*hist | ncclimo ...
```
http://nco.sf.net/nco.html#ncclimo