Due to the current gap in continued funding from the U.S. National Science Foundation (NSF), the NSF Unidata Program Center has temporarily paused most operations. See NSF Unidata Pause in Most Operations for details.
The netCDF Operators NCO version 4.9.0 are woke. http://nco.sf.net (Homepage, Mailing lists, Help) http://github.com/nco (Source Code, Issues, Releases, Developers) What's new? Version 4.9.0 includes new regridder features, JSON, and ncap2 fixes. Notably, this version simplifies the weight-application interface, parallelizes ncremap in stand-alone mode, and completely revamps, improves, and accelerates the sub-gridscale weight application algorithm. The NCO conservative weight-generator algorithm also debuts. Previously, ncremap always used ESMF or TempestRemap to make weights. Use 'ncremap -a nco' to get NCO's conservative weight generation. NCO will always support ESMF and TR which are more time-tested. However, NCO has some unique features, like an option to output the overlap mesh in SCRIP format, and it performs best for some grids. Work on NCO 4.9.1 has commenced and will improve NCO weight-generator accuracy, reduce vertical interpolation memory use, and supply more accurate weight generation options for rectangular lat-lon grids. Work on NCO 5.0.0 has commenced "under the hood". A key leap in that release will be support for netCDF4 user-defined types. Printing of netCDF4 user-defined types ENUM and VLEN is ready now (though unsupported) with the --udt flag. 5.0.0 will contain the finished version of that, and include options for invoking mbtempest in place of tempest. Enjoy, Charlie NEW FEATURES (full details always in ChangeLog): A. ncremap has been refactored for full parallelism. Previously the standalone ncremap could only run parallelized when invoked with pure weight application requests. Special processing like CLM/ELM sub-gridscale regridding or MPAS juggling only worked with single file invocations (which is what ncclimo always requests). Now the standalone ncremap is parallelized and adheres to --job_nbr control for all regridding in standalone invocations, meaning that regridding lists of files offline with MPI or background parallelism is now much quicker: ls cam/*.nc | ncremap --job_nbr=4 --par_typ=mpi -m map.nc ls mpas/*.nc | ncremap -P mpas -j 4 -p mpi -m map.nc ls elm/*.nc | ncremap --sgs_frc=landfrac -j 16 -m map.nc http://nco.sf.net/nco.html#ncremap B. ncremap has a new API for sub-gridscale (SGS) regridding. Previously ncremap employed a convoluted and slow procedure to remap data with sub-gridscale fractions, e.g., CLM/ELM, CICE, and MPAS-Seaice data. That procedure and ncremap API used to require specification of the source and destination grids. The new, faster ncremap SGS procedure requires only the map-file and the name of the sub-gridscale fraction field. The -P sgs option is no longer required, thought is still accepted for back-compatibility. ncremap --sgs_frc=landfrac -m map.nc # CLM/ELM ncremap --sgs_frc=frc.nc/landfrac -m map.nc # CLM/ELM external ncremap -P elm -m map.nc # CLM/ELM alternative ncremap --sgs_frc=aice --sgs_msk=tmask -m map.nc # CICE ncremap -P cice -m map.nc # CICE alternative ncremap --sgs_frc=timeMonthly_avg_iceAreaCell -m map.nc # MPAS-Seaice ncremap -P mpasseaice -m map.nc # MPAS-Seaice alternative http://nco.sf.net/nco.html#ncremap C. ncremap supports the new NCO weight generation algorithm. The NCO algorithm is first-order conservative and compares well in accuracy with other FV algorithms such as ERWG and TR. Special thanks to Henry Butowsky for his dedication to this. We evaluated ERWG "conserve", NCO "nco", and TR "mono" with --chk_map (see below) on the E3SMv1 lo-res and hi-res grids. The three algorithms generally agree to 10-13 significant digits. ESMF won for atm->atm remapping, and NCO for ocn->atm remapping, both by razor-thin margins. YMMV. We hope users will try the NCO algorithm and send feedback. NCO weight-generation is threaded and scales well to 3-4 threads. ncremap -a nco -s grd_src.nc -d grd_dst.nc -m map.nc in.nc out.nc ncremap -t 4 -a nco -s grd_src.nc -d grd_dst.nc -m map.nc http://nco.sf.net/nco.html#ncremap D. ncks supports a new option --chk_map to evaluate the quality of regridding weights. This option works with map-files (not grid-files) in ESMF/CMIP6-compliant format (i.e., sparse matrix S and coordinates [xy][ab]_[cv]). When invoked with the additional --area_wgt option (also new), the evaluation statistics are area-weighted and thus reflect exactly the global-mean/min/max/ mebs/rms/sdn biases expected when regridding globally uniform fields. This tool makes it easier to objectively assess weight-generation algorithms, and will hopefully assist in their improvement. ncks --chk_map map.nc ncks --chk_map --area_wgt map.nc http://nco.sf.net/nco.html#chk_map http://nco.sf.net/nco.html#area_wgt E. ncremap works with newer MPAS Ocean files that contain BGC dimensions like R3 and FOUR. ncremap -P mpas -m map_bilin.nc in.nc out.nc http://nco.sf.net/nco.html#ncremap F. ncremap now propagates all netCDF formats except CDF5 to TempestRemap (TR) (which does not yet accept or produce CDF5 files AFAICT). http://nco.sf.net/nco.html#tr G. ncremap omits the TR --volumetric flag from fv2se_stt maps. TempestRemap author Paul Ullrich recommends this to keep fv2se_stt maps consistent (producing output in same range) with fv2se_flx maps. ncremap -a fv2se_stt -s grd_src.nc -d grd_dst.nc -m map.nc in.nc out.nc http://nco.sf.net/nco.html#ncremap H. ncremap allows simultaneous invocation of horizontal and vertical regridding. Yes, now ncremap can automagically regrid any file to any new horizontal and vertical grid at the same time. Simply supply both a map-file and a vertical grid-file: ncremap -m map.nc --vrt=vrt_grd.nc in.nc out.nc http://nco.sf.net/nco.html#vrt http://nco.sf.net/nco.html#ncremap I. ncremap/ncclimo have deprecated support for options named for the short-lived ALM land surface model. Use 'elm' instead of 'alm', ncremap -P elm -m map.nc http://nco.sf.net/nco.html#ncremap J. NCO supports flexible options to specify non-default names for vertical grid coordinates in input and output (vertically interpolated) data files. ncks --rgr plev_nm=vrt_nm --vrt=vrt_grd.nc in.nc out.nc ncremap -n '--rgr plev_nm=vrt_nm' --vrt=vrt_grd.nc in.nc out.nc http://nco.sf.net/nco.html#lev_nm K. ncks supports a new option --chk_nan to quickly find the location of any NaN values in a dataset. Thanks to Matthew Thompson of NASA for this suggestion. ncks --chk_nan in.nc http://nco.sf.net/nco.html#chk_nan BUG FIXES: A. ncap2 --output again works with single command-line files This was inadvertently broken in 4.8.0. Thanks to Kyle Wilcox for reporting this issue. B. Regridder allows "none" for SCRIP normalization type for bilinear regridding, employed in UKMO SCRIP files. Thanks to Craig MacLachlan for providing this fix. C. ncremap vertical interpolation bug-fix for fields stored on interface levels of hybrid coordinate grids (e.g., CMFMC). D. ncremap bug-fix for pressure-to-pressure grid vertical interpolation for input files with no time dimension. E. ncks --json fixes JSON output of NC_FLOAT and NC_DOUBLE by eliminating the trailing '.' (illegal in JSON) from integer-valued floats. '1234567.' is now output as '1234567'. Thanks to Aleksandar Jelenak for this bug report. F. ncremap no longer automatically employs mean-preserving algorithm when encountering missing_data. Thanks to Xylar Asay-Davis for patiently suggesting saner behavior. Full release statement at http://nco.sf.net/ANNOUNCE -- Charlie Zender, Earth System Sci. & Computer Sci. University of California, Irvine 949-891-2429 )'(
netcdfgroup
archives: