The Unidata Program Center will be altering the IDD NIMAGE datastream to contain all of the GOES-16 and GOES-17 Level 2 image products that are currently being being sent in the IDD NOTHER and HDS datastreams.
We plan to begin distributing a reconstituted NIMAGE feed
from our top level IDD relay clusters (idd.unidata.ucar.edu and
iddb.unidata.ucar.edu) in the mid-June time frame. (Note: We will postpone
turning on this feed if/when end users indicate that they need more
time to prepare for the change.)
This is important because the volume of data available in the NIMAGE feed will increase from an average of about 100 MB/hr to an average that is close to 7 GB/hr, a 70-fold increase!
Why is Unidata Making This Change?
NOAA has indicated that operation of GOES-15 will likely cease in early-mid July:
https://www.goes-r.gov/users/transitionToOperations17.html
Since GOES-East imagery in GINI format was removed from the NOAAPort SBN after GOES-13 was replaced by GOES-16 and then decommissioned, we believe that it is most likely that all GINI imagery will cease to be distributed in NOAAPort in the mid-July time frame. Since the NIMAGE IDD data feed has been populated solely by the GINI imagery distributed in NOAAPort, this feed will cease to contain any data.
Currently, NOAAPort contains GOES-16 and GOES-17 ABI (Advanced Baseline Imager) Level 2 product tiles (pieces of full image scenes), and those tiles are being distributed in the IDD NOTHER data feed. To date, these image tiles would need to be stitched together by end users before they were fully usable in display and analysis applications available from Unidata (e.g., AWIPS, GEMPAK, IDV, McIDAS and MetPy).
In addition to the ABI Level 2 product tiles, 19 other Level 2 products are also being distributed in NOAAPort. Unidata has been making those products available in the IDD NOTHER and HDS data feeds since their addition. Like the ABI Level 2 product tiles, the other Level 2 products need some processing before they are usable in end user applications — they need to have the NOAAPort header and footer stripped off to leave the underlying netCDF4 files.
We believe that it will be much easier for end users who get their NOAAPort-delivered Level 2 images and products via the IDD if we reconstitute full images scenes from ABI tiles and strip NOAAPort broadcast headers and footers here at the Unidata Program Center and then distribute the resulting netCDF4 images/products in a reconstituted NIMAGE IDD data feed. Also, reconstituting the NIMAGE feed will allow us to make other, value-added GOES-16/17 products available to the community. The first of these products is Geostationary Lightning Mapper (GLM) imagery/grids being produced at Texas Tech by community member Eric Bruning.
Information on the imagery and products that are currently contained in and those that will be added to the reconstituted NIMAGE feed can be found in:
https://www.unidata.ucar.edu/data/nimage.html
We have been using all of the products in this new incarnation of the NIMAGE feed internally for the past few months to test their viability in the various application packages that we provide and support. We are satisfied that transition to use of these products by end users will be straightforward, but it will require that sites install and transition to use of the latest of each package that we make available.
Important Things to Note
1. The reconstitution of the NIMAGE feed comes at a cost
The volume of data available will increase from an average of about 100 MB/hr to an average nearing 7 GB/hr, a 70-fold increase!
Sites that do not want to receive any/all of the processed GOES-16/17 imagery/products in the new NIMAGE feed need to make sure that their LDM configuration file REQUEST(s) do not include products that they do not want.
The current GOES-15 GINI images being distributed in NOAAPort all have LDM/IDD Product IDs that begin with 'satz'. Sites requesting all of the current NIMAGE feed via a REQUEST that looks like:
REQUEST NIMAGE ".*" idd.unidata.ucar.edu
and that either do not want or can not handle the increased data volume should immediately change their REQUEST to something like:
REQUEST NIMAGE "satz" idd.unidata.ucar.edu
Sites that do want to receive all of the imagery that will be available in the reconstituted NIMAGE feed and whose network connection can handle the increased volume can leave their REQUEST(s) as is, but this decision should not be taken lightly given the 70-fold increase in data volume.
Sites wanting to receive some of the data from the reconstituted NIMAGE feed are advised to read through the NIMAGE web page listed above to become familiar with how the LDM/IDD Product IDs will look and then re-fashion their feed REQUEST(s) to get the set of products desired.
In addition, sites wanting to use some/all of the data in the new incarnation of the NIMAGE feed will need to modify/add LDM pattern-action file action(s) to store the products for use in the display and analysis package(s) that they use. The NIMAGE web page contains links to example pattern-action file actions that illustrate one approach for supporting the new imagery. Sites that have questions about how they should modify their existing LDM setup should contact Unidata User Support for assistance (see below).
2. Sites must upgrade visualization applications
Sites that want to use the products that will be available in the reconstituted NIMAGE feed should immediately install and switch to using the most recent releases of their Unidata-supplied display and analysis applications
This is especially true for GEMPAK users, as imagery in netCDF4 format was not supported in Unidata GEMPAK prior to the current release.
3. Data volumes are being minimized
The volume of the data that will be available in the reconstituted NIMAGE feed is actually 20% less than the volume of the sum of the products being delivered in NOAAPort and distributed in the NOTHER and HDS datastreams. The reason for this is twofold:
- NOAAPort broadcast headers and footers have been removed resulting in a minor reduction of volume
- netCDF4 compression has been implemented/implemented better in the products
In Summary
The reconstituted IDD NIMAGE datastream will contain:
- Full image scenes that have been created by stitching together the ABI tiles sent in the NOAAPort SBN and distributed in the IDD NOTHER feed
- All other Level 2 products being distributed in the NOAAPort SBN in the IDD NOTHER and HDS feeds
- Value-added GLM image/grid products created by Eric Bruning of Texas Tech University
Note also:
- All products are in netCDF-4 format
- All products are in the GOES-R mission standard Fixed Grid projection
- The names for all products (contained in the LDM/IDD Product IDs)
follow the GOES-R mission standard naming convention as detailed
in the
GOES-R SERIES PRODUCT DEFINITION AND USERS’ GUIDE (PUG). - In all cases, the LDM/IDD Product IDs will consist of fully qualified path names for products, and the file name portion of the fully qualified names follow GOES-R mission standards for naming. The NIMAGE web page referenced above lists example Product IDs for each product.
Please send questions and comments about the change to the IDD NIMAGE feed to: support-datastream@unidata.ucar.edu
