« NetCDF operators... | Main

03 April 2025

Version 5.3.3 of the netCDF Operators (NCO) has been released. NCO is an Open Source package that consists of a dozen standalone, command-line programs that take netCDF files as input, then operate (e.g., derive new data, average, print, hyperslab, manipulate metadata) and output the results to screen or files in text, binary, or netCDF formats.

The NCO project is coordinated by Professor Charlie Zender of the Department of Earth System Science, University of California, Irvine. More information about the project, along with binary and source downloads, are available on the SourceForge project page.

From the release message:

Version 5.3.3 adds IEEE floating-point exception testing to ncks. All operators include corner-case quantization fixes for strange numbers (-0.0, NaN) that are also now in the netCDF-C Library. ncclimo now only tries to create regional timeseries statistics on variables that are single-level horizontal lat/lon/ncol fields. ncremap supports interpolation of pure-pressure to hybrid sigma-pressure coordinates with distinct source and destination surface pressure fields. And ncrename adds some words of caution. Skip this release if these changes do not interest you.

Did you realize that the first public release of ncks was May 2, 1995? https://www.unidata.ucar.edu/mailing_lists/archives/netcdfgroup/1995/msg00069.html Happy 30th birthday ncks!!!

New Features

The new ncks --fpe option reports the status of all IEEE Floating Point Exceptions (FPEs) after execution. This can be useful to verify whether input datasets contain strange numbers that could be arithmetically processed, and to ensure the integrity of NCO arithmetic.

% ncks --fpe -O -7 -L 1 -C -v ppc_zro_ngt_nan_flt --qnt_alg=gbr \
        --qnt default=3 --cmp='shf|zst' in.nc out.nc
ncks: Successfully set FE_INVALID|FE_DIVBYZERO|FE_OVERFLOW exception flags
ncks: FE_DIVBYZERO is not raised
ncks: FE_INEXACT is raised
ncks: FE_INVALID is not raised
ncks: FE_OVERFLOW is not raised
ncks: FE_UNDERFLOW is not raised

http://nco.sf.net/nco.html#fpe
http://nco.sf.net/nco.html#ncks

All operators can now quantize -0.0 (a legal number) without triggering any IEEE Floating Point Exceptions (FPEs). Other weird numbers, like NaN, already avoided FPEs. Moreover, the necessary changes to avoid FPEs have been merged upstream into the quantization algorithms in libnetcdf version 4.10-development.
http://nco.sf.net/nco.html#qnt
When no explicit variable extraction list is specified in timeseries mode, ncclimo automatically creates timeseries for all input variables with at least two dimension. However, when regional statistics are also requested with --rgn_stt, these statistics are only meaningful for spatial timeseries, i.e., single level fields with latitude, longitude, and time. Under these circumstances ncclimo now only creates regional statistics for single level spatial timeseries. Previously it would try (and fail) to generate regional statistics for multi-level or otherwise inappropriate rank >=2 variables.
```
ls caseid*.nc | ncclimo --split --rgn_avg -s 2013 -e 2014 -o foo
```
http://nco.sf.net/nco.html#ncclimo
Previous versions of NCO could only interpolate ECMWF/IFS data to hybrid sigma-pressure coordinates when the input surface pressure (lnsp) was for the output as well. Now NCO can interpolate to a distinct output surface pressure that is storeed in the vertical template file (a so-called "fat" template that includes the surface pressure, rather than a "skinny" template that does not).
```
ncremap --ps_nm=lnsp --vrt_fl=vrt.nc in.nc out.nc
```
http://nco.sf.net/nco.html#vrt
http://nco.sf.net/nco.html#ncremap
ncks --chk_map now prints the minimum and maximum angular and horizontal resolution of the source and destination grids. The angular resolution is estimated as the square-root of the area (in sr) of the smallest/largest gridcells, and is expressed in degrees. The spatial resolution is estimated. This is the angle subtended by a spherical cap with the same area as the gridcell.
```
% ncks --chk_map map_ne30pg2_to_cmip6_180x360_ncoaave.20231201.nc
...
area_a min, ~dx: 4.9093515660647374e-04 sr, ~141.14 km, ~1.27 degrees in
grid A cell [14462,-45.7267,+177.69]
area_a max, ~dx: 6.8523331840188574e-04 sr, ~166.75 km, ~1.50 degrees in
grid A cell [1858,+0.749936,+359.25]
...
area_b min, ~dx: 2.6580860638549116e-06 sr, ~10.39 km, ~0.09 degrees in
grid B cell [60,-89.5,+59.5]
area_b max, ~dx: 3.0460968486221404e-04 sr, ~111.18 km, ~1.50 degrees in
grid B cell [32272,-0.5,+231.5]
...
```
This feature can be useful in assessing the uniformity of grids, and
http://nco.sf.net/nco.html#chk_map
http://nco.sf.net/nco.html#ncks

ncrename now WARNs users attempting to rename coordinates in netCDF4 files. The WARNING, which prints only when debugging level > 0, the input file is in netCDF4 format, and the user tries to rename a variable or dimension, is self-explanatory:

zender@spectral:~$ ncrename -D 1 -O -v lat,latitude in.nc out.nc
ncrename: WARNING Renaming variables, dimensions, and attributes in any
netCDF3-format file (i.e., classic, 64-bit, and CDF5) ALWAYS works.
Moreover, renaming groups and attributes in any netCDF4-format file
always works. However, these re-assuring notes precede a discomfitting
message: A longstanding problem in the netCDF library (it is not an NCO
issue per se) sometimes crashes ncrename or, even worse, silently
corrupts data in the output file when input file is netCDF4-format.
These outcomes are only known to occur when attempting to rename
variables to/from the same names as dimensions (i.e., coordinate
variables), and or dimensions to/from the same names as variables. A
long thread describing the status of this issue since 2017 is at
https://github.com/Unidata/netcdf-c/issues/597. A chronology of the
issue dating back to 2007 is at
https://nco.sourceforge.net/nco.html#bug_nc4_rename.
ncrename HINT: Do not expect this issue to be fixed in the near future :)
ncrename HINT: There is a straightforward two- or three-step workaround
that ALWAYS works: 1. Translate the file to any netCDF3 format (e.g.,
with "ncks -5 in.nc out.nc"). 2. Perform the renaming. 3. (Optional,
only necessary if you wish to recover the netCDF4 features like
compression) Translate back to a netCDF4-format (e.g., with "ncks -7
in.nc out.nc").
ncrename: In total renamed 0 attributes, 0 dimensions, 0 groups, and 1
variable
zender@spectral:~$ ncrename -O -v lat,latitude in.nc out.nc
zender@spectral:~$ ncrename -D 1 -O -a lat@units,meaning in.nc out.nc
ncrename: In total renamed 1 attribute, 0 dimensions, 0 groups, and 0
variables
zender@spectral:~$ ncrename -D 1 -O -v lat,latitude in3.nc out.nc
zender@spectral:~$

The second and third and fourth examples show no WARNING output because the debugging level is the default (0) in the second, and because the user is not renaming a variable or dimension in the third, and because the input file is netCDF3-format in the fourth. The decision was made to turn-on this WARNING only in a non-default debug level to avoid annoying users with existing workflows that expect no text output from ncrename. This compromise is intended to prevent widespread panic and mass hysteria.

Additional details are available in the ChangeLog.

Posted by Unidata News [ Comments [0] ]