« ESIP: Community... | Main | Unidata is looking... »

09 September 2020

Version 4.9.4 of the netCDF Operators (NCO) has been released. NCO is an Open Source package that consists of a dozen standalone, command-line programs that take netCDF files as input, then operate (e.g., derive new data, average, print, hyperslab, manipulate metadata) and output the results to screen or files in text, binary, or netCDF formats.

The NCO project is coordinated by Professor Charlie Zender of the Department of Earth System Science, University of California, Irvine. More information about the project, along with binary and source downloads, are available on the SourceForge project page.

From the release message:

Version 4.9.4 contains new features focused on de-interleaving time coordinates and per-record weighting for ncra, high-freqency (e.g., diurnally-resolved) climatologies and new defaults for ncclimo, a new distance-weight extrapolation algorithm for ncremap, per-file weights for nces. New NCO-wide features include: support for unbuffered I/O that can speed-up I/O for large record variables, more precise quantization than Bit Grooming, and faster arithmetic that takes advantage of SIMD directives on OpenMP-enabled builds. Overall release is packed with interesting new features...read on.

New Features

NCO may now be configured with --enable-gpu at build-time to offload certain arithmetically intensive computations to the GPUs with select architectures and compilers. This feature currently has no speed benefits, and needs a volunteer to lead development. Please contact me if interested.
All operators now support unbuffered I/O with netCDF3 files when invoked with the flag --uio or longer synonyms --unbuffered_io or --share_all. This flag invokes the netCDF library NC_SHARE flag which enables unbuffered (non-cached) I/O. Unbufferend I/O may significantly reduce throughput time when large record variables are written or read. Performance improvements may depend on netCDF version. Thanks to Barron Henderson for this suggestion.
```
ncks -v T in.nc out.nc # Default, buffered I/O
ncks --uio -v T in.nc out.nc # Unbuffered I/O
ncra --uio -v T,Q,U,V in*.nc out.nc
```
http://nco.sf.net/nco.html#uio
The default quantization method has changed from Bit Grooming to Bit Rounding, contributed by Rostislav Kouznetsov with suggestions from Milan Klower. As implemented, Bit Rounding will substantially improve the precision for the specified number of significant digits to retain. A future version of NCO will re-tune the number of bits per retained digit to turn this precision advantage into a compression advantage. An article submitted by R. Kouznetsov to GMD describes Bit Rounding more...precisely.
http://nco.sf.net/nco.html#bg
The multi-file, multi-record operators, ncra and ncrcat now support interleaved time-coordinates in groups of records. Interleaving (or de-interleaving, depending on one's perspective) means altering the order of records in a group to be processed. Specifically, the interleaving feature causes the operator to treat as sequential records those that are separated by multiples of the specified interleave parameter within a group of records. Specify the interleave parameter as the fifth hyperslab argument. The interleave feature sequences records with respect to their position relative to the beginning of each sub-cycle. Records a multiple of interleave from sub-cycle beginning are first extracted (by ncrcat) or reduced (by ncra), then records offset from these by one, two, et cetera up to interleave-1. Thus interleaving allows deconvolution of periodic phenomena within a time-series. Some examples to reify the abstract:

Let in1.nc = [1..10], in2.nc = [11..20], and in12.nc = [1..20].
```
ncra -d time,,,,10,5 in1.nc ~/foo.nc # 3.5, 4.5, 5.5, 6.5, 7.5
ncrcat -d time,0,4,,6,2 in1.nc ~/foo.nc 1, 3, 5, 2, 4, 6 (+WARNING)
ncrcat -d time,2,,10,4,2 in12.nc ~/foo.nc # 3, 5, 4, 6, 13, 15, 14, 16
ncra   -d time,2,,10,4,2 in12.nc ~/foo.nc # 4, 5, 14, 15
ncra -d time,,,,10,2 in1.nc in2.nc ~/foo.nc # 5, 6, 15, 16
ncra -d time,,,,10,2 in12.nc ~/foo.nc # 5, 6, 15, 16
```
http://nco.sf.net/nco.html#interleave
http://nco.sf.net/nco.html#ilv
nces now supports the -w (or --wgt) option for per-file weights. This option is similar to the ncra --wgt option. The nces version also accepts a variable name that contains a scalar per-file-weight. Per-file weights are useful when computing statistics of ensembles whose members should be weighted unevenly. Hence these three commands produce the same answers, though the second and third are much more flexible and can have non-integral weights:
```
nces   1.nc 2.nc 2.nc out.nc
nces -w 1,2 1.nc 2.nc out.nc
nces -w var 1.nc 2.nc out.nc
```
http://nco.sf.net/nco.html#nces
http://nco.sf.net/nco.html#xmp_nces
ncra now supports the --per_record_weights (or --prw) option to utilize command-line weights specified by -w (or --wgt) for per-record weights instead of per-file-weights. This is useful when computing weighted averages with cyclically varying weights, since the weight given on the command line will be repeated for the length of the timeseries. Consider, for example, a CMIP6 timeseries of historical monthly mean emissions that one wishes to convert to an timeseries of annual-mean emissions. One can weight each month by its number of days via:
```
ncra --per_record_weights --mro -d time,,,12,12 --wgt \
31,28,31,30,31,30,31,31,30,31,30,31 ~/monthly.nc ~/annual.nc
```
Thanks to Philip Cameron-Smith of LLNL for this suggestion.
http://nco.sf.net/nco.html#ncra
http://nco.sf.net/nco.html#xmp_ncra
http://nco.sf.net/nco.html#prw
ncra accepts the new flag --promote_ints (or --prm_ints) to output statistics of integer-valued input variables in floating-point precision in the output file. By default, NCO arithmetic operators such as ncra auto-promote integers to double-precision prior to arithmetic, then conduct the arithmetic, then demote the values back to integers for final output. This default behavior quantizes the mantissa of the values and prevents, e.g., turning statistical means of boolean (0 or 1-valued) input data into floating point probabilities. The --promote_ints flag causes the statistical means of integer (including NC_BYTE) inputs to be output as single-precision floating point (NC_FLOAT) variables. This allows use arithmetic to be performed on Boolean values stored in the space-conserving NC_BYTE (single byte) format in input files.
```
ncra --prm_ints in*.nc out.nc
```
Thanks to Paul Ullrich of UC Davis for this suggestion.
http://nco.sf.net/nco.html#prm_ints
http://nco.sf.net/nco.html#promote_ints
ncremap understands new dimensions used in DOE E3SM MPAS BGC simulations. ncremap also supports the new --pdq_opt to override internal presets and to future-proof itself against unexpected new dimensions from any model input.
```
ncremap -P mpasseaice --map=map.nc in.nc out.nc
ncremap --pdq='-a Time,new_dim,nCells'  --map=map.nc in.nc out.nc
ncremap --pdq='-a time,new_dim,lat,lon' --map=map.nc in.nc out.nc
```
Thanks to Ahmed Elshall for reporting the new dimensions.
http://nco.sf.net/nco.html#pdq_opt
ncremap allows access to a new regridding algorithm based on distance-weighted extrapolation (DWE). DWE is similar to the ESMF nearestidavg extrapolation alorithm, and accepts the same two parameters as input: --xtr_xpn sets the (absolute value of) the exponent used in distance weighting (default is 2.0), and --xtr_nsp sets the number of source points used in the extrapolation (default is 8). ncremap can apply DWE to the entire destination grid, or just to points with missing/masked values.
```
ncremap --alg_typ=nco_dwe -s src.nc -d dst.nc -m map.nc
ncremap -a nco_dwe --xtr_xpn=1.0 -s src.nc -d dst.nc -m map.nc
ncremap -a nco_dwe --xtr_nsp=1   -s src.nc -d dst.nc -m map.nc
```
Thanks to Henry Butowsky for implementing the new method.
http://nco.sf.net/nco.html#dwe

The ncks --dt_fmt option now applies equally well to JSON and XML output as to CDL output:

% ncks -d time,0 -v time --cdl --dt_fmt=3 ~/nco/data/in.nc
...
time = "1964-03-13T21:09:0.000000" ;
...
% ncks -d time,0 -v time --json --dt_fmt=3 ~/nco/data/in.nc
...
"data": ["1964-03-13T21:09:0.000000"]
...
% ncks -d time,0 -v time --xml --dt_fmt=3 ~/nco/data/in.nc
...
<ncml:values separator="*">1964-03-13T21:09:0.000000</ncml:values>
...

Thanks to Troy Mare for this suggestion.
http://nco.sf.net/nco.html#dt_fmt
http://nco.sf.net/nco.html#json
http://nco.sf.net/nco.html#xml

ncra, nces, and ncrcat introduce the --clm_nfo (or --cb) option to produce CF-conformant climatological times and bounds. This option takes a comma-separated argument list of five relevant input parameters: --cb=yr_srt,yr_end,mth_srt,mth_end,tpd, where yr_srt is the climatology start-year, yr_end is the climatology end-year, mth_srt is the climatology start-month (in [1..12] format), mth_end is the climatology end-month (in [1..12] format), and tpd is the number of timestpes per day (with the special exception that tpd=0 indicates monthly data, not diurnally-resolved data. A seasonal summer climatology created from monthly mean input data spanning June, 2000 to August, 2020 should call ncra with --clm_bnd=2000,2020,6,8,0, whereas a diurnally resolved climatology of the same period with 6-hourly input data resolution would use --clm_bnd=2000,2020,6,8,4.
```
ncra --cb=2014,2016,1,1,0 2014_01.nc 2015_01.nc 2016_01.nc clm_JAN.nc
```
http://nco.sf.net/nco.html#cb
http://nco.sf.net/nco.html#ncra
ncclimo has changed default settings for two parameters. As of this version, ncclimo sets the options "-a sdd --no_amwg_links" by default. For seasonally contiguous DJF climos one must now explicitly set "-a scd". To create symbolic links to climatology files with AMWG names, one must now explicitly request --amwg_links.
http://nco.sf.net/nco.html#dec_md
http://nco.sf.net/nco.html#lnk_flg
ncclimo now supports the high-frequency climos and splitting. Access these capabilities by specifying the climatology-mode options hfc and hfs, respectively. In both cases (climos and splitting) the input file(s) name will not be constructed automatically and must be provided via stdin, positional command-line arguments, or a director to glob. For climos, ncclimo will detect the number of timesteps per day (tpd) in the input data, and compute the climatological mean diurnal cycle from the input data. The output is similar to monthly climos, except each climatological monthly, seasonal, or annual output file will contain tpd timesteps to represent the diurnal cycle.
```
# Split high-frequency timeseries into CMIP-like timeseries
cd ;ls *.h4.nc | ncclimo --clm_md=hfs -v=T \
--ypf=1 --yr_srt=56 --yr_end=76 --drc_out=
# Generate diurnal climos from high-frequency CMIP6 timeseries
cd ;ls *.h4.nc | ncclimo --clm_md=hfc
-c  --yr_srt=2001 --yr_end=2002 --drc_out=
```
http://nco.sf.net/nco.html#ncclimo
http://nco.sf.net/nco.html#clm_md
ncclimo now outputs more CF-conformant climatological times and bounds for all climatologies. Previously, ncclimo output a time-centered value for climatological bounds, now it outputs an initial YYYYMMDD format, as recommended by CF examples such as http://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-conventions.html#climatological-statistics Example 7.13.