IMPORTANT notice of major changes to the IDD NIMAGE datastream; please read.
First, we apologies for cross posting this announcement to multiple lists,
but we want to make sure that all users that will/could be affected
are notified.
Second, we realize that the following message is long, but please read
it all since it contains important information that will affect sites
that are or will be receiving imagery products in the IDD NIMAGE feed.
What:
We will be transitioning the IDD NIMAGE datastream to one that
contains all of the GOES-16 and GOES-17 Level 2 image products
that are currently being being sent in the IDD NOTHER and HDS
datastreams.
When:
It is our desire to begin distributing a reconstituted NIMAGE feed
from our top level IDD relay clusters (idd.unidata.ucar.edu and
iddb.unidata.ucar.edu) in the mid-June time frame. We will postpone
turning on this feed if/when end-users indicate that they need more
time to prepare for the change.
Impact:
The volume of data available in the NIMAGE feed will increase from
an average of about 100 MB/hr to an average that is close to 7 GB/hr,
a 70-fold increase!
Reason for the change:
NOAA has indicated that operation of GOES-15 will likely cease in
early-mid July:
https://www.goes-r.gov/users/transitionToOperations17.html
Since GOES-East imagery in GINI format was removed from the NOAAPort
SBN after GOES-13 was replaced as GOES-East by GOES-16 and then
decommissioned, we believe that it is most likely that all GINI imagery
will cease to be distributed in NOAAPort in the mid-July time frame.
Since the NIMAGE IDD data feed has been populated solely by the GINI
imagery distributed in NOAAPort, this feed will cease to contain any data.
Currently, NOAAPort contains GOES-16 and GOES-17 ABI (Advanced Baseline
Imager) Level 2 product tiles (pieces of full image scenes), and those
tiles are being distributed in the IDD NOTHER data feed. To date, these
image tiles would need to be stitched together by end-users before they
were fully usable in display and analysis applications available from
Unidata (e.g., AWIPS, GEMPAK, IDV, McIDAS and MetPy (this is not
strictly true for AWIPS)).
In addition to the ABI Level 2 product tiles, 19 other Level 2 products
are also being distributed in NOAAPort, and we have been making those
products available in the IDD NOTHER and HDS data feeds since their
addition to NOAAPort. Like the ABI Level 2 product tiles, the other
Level 2 products need some processing before they are usable in end-user
applications - they need to have the NOAAPort header and footer stripped
off to leave the underlying netCDF4 files.
We believe that it will be much easier for end-users, who get their
NOAAPort-delivered Level 2 images and products via the IDD, if we
reconstitute full images scenes from ABI tiles and strip NOAAPort
broadcast headers and footers here in Unidata and then distributed the
resulting netCDF4 images/products in a reconstituted NIMAGE IDD data
feed. Also, reconstituting the NIMAGE feed will allow us to make other,
value-added GOES-16/17 products available to the community. The first
of these products is GLM imagery/grids being produced at Texas Tech by
community member Eric Bruning.
Information on the imagery and products that are currently contained
in and those that will be added to the reconstituted NIMAGE feed can be
found in:
https://www.unidata.ucar.edu/data/nimage.html
We have been using all of the products in this new incarnation of the
NIMAGE feed internally for the past few months to test their viability
in the various application packages that we provide and support. We are
satisfied that transition to use of these products by end users will be
straightforward, but it will require that sites install and transition
to use of the latest of each package that we make available.
Important things to note:
- the reconstitution of the NIMAGE feed comes at a cost
The volume of data available will increase from an average of about
100 MB/hr to an average nearing 7 GB/hr, a 70-fold increase!
Sites that do not want to receive any/all of the processed GOES-16/17
imagery/products in the new NIMAGE feed need to make sure that their
LDM configuration file REQUEST(s) do not include products that they
do not want.
The current GOES-15 GINI imagery that is being distributed in NOAAPort
all have LDM/IDD Product IDs that begin with 'satz'. Sites requesting
all of the current NIMAGE feed via a REQUEST that looks like:
REQUEST NIMAGE ".*" idd.unidata.ucar.edu
and that either do not want or can not handle the increased data volume
should immediately change their REQUEST to something like:
REQUEST NIMAGE "satz" idd.unidata.ucar.edu
Sites that do want to receive all of the imagery that will be available
in the reconstituted NIMAGE feed and whose network connection can handle
the increased volume can leave their REQUEST(s) as is, but this decision
should not be taken lightly given the 70-fold increase in data volume!
Sites wanting to receive some of the data from the reconstituted
NIMAGE feed are advised to read through the NIMAGE web page listed
above to become familiar with how the LDM/IDD Product IDs will look
and then re-fashion their feed REQUEST(s) to get the set of products
desired.
- sites that want to use the products that will be available in the
reconstituted NIMAGE feed should immediately install and switch to
using the most recent releases of their Unidata-supplied display and
analysis applications
This is especially true for GEMPAK users as imagery in netCDF4 format
was not supported in Unidata GEMPAK prior to the current release.
- the data volume that will be available in the reconstituted NIMAGE
feed is actually 20% less than the volume of the sum of the products
being delivered in NOAAPort and distributed in the NOTHER and HDS
datastreams
The reason for this is twofold:
- NOAAPort broadcast headers and footers have been removed
resulting in a minor reduction of volume
- netCDF4 compression has been implemented/implemented better in
the products
Recap:
The reconstituted IDD NIMAGE datastream will contain:
- full image scenes that have been created by stitching together
the ABI tiles sent in the NOAAPort SBN and distributed in the
IDD NOTHER feed
- all other Level 2 products being distributed in the NOAAPort SBN
in the IDD NOTHER and HDS feeds
- value added GLM image/grid products created by Eric Bruning
of Texas Tech University
- all products are in netCDF-4 format
- all products are in the GOES-R mission standard Fixed Grid
projection
- the names for all products (contained in the LDM/IDD Product IDs)
follow the GOES-R mission standard naming convention as detailed
in:
GOES-R SERIES PRODUCT DEFINITION AND USERS’ GUIDE (PUG)
https://www.goes-r.gov/users/docs/PUG-GRB-vol4.pdf
In all cases, the LDM/IDD Product IDs will consist of fully qualified
path names for products, and the file name portion of the fully
qualified names follow GOES-R mission standards for naming. The NIMAGE
web page referenced above lists example Product IDs for each product.
Please send any/all questions/comments about the upcoming change to the
IDD NIMAGE feed to:
support-datastream@xxxxxxxxxxxxxxxx
Best regards,
Unidata User Support
--
********************************************************************** <
Unidata User Support UCAR Unidata Program <
(303)497-8643 P.O. Box 3000 <
support@xxxxxxxxxxxxxxxx Boulder, CO 80307 <
---------------------------------------------------------------------- <
Unidata Web Support http://www.unidata.ucar.edu/support <
********************************************************************** <
