Hello Tom,
I have questions about this transition, and I'm confused about a few
things, so hopefully this can be cleared up.
First: Does this affect anything on the SATELLITE/nee DIFAX feed? That is
strictly for data coming across the GRB feed and will stay there, correct?
Second: What about those who have their own ingest dishes? If they use IDD
as a backup, will they need to have a duplicate feed coming across: one
from their dish on NOTHER/EXP, and one from UNIDATA? If so...
Third: Does this require an LDM or LDM database upgrade, the latter of
which can be done automagically in the latest version, 6.13.11?
Fourth: Can the LDM switch the NOAAport-ingested GOES products over to
NIMAGE as well to avoid duplicating the UNIDATA feed on the current
feedtypes IF a site has their own dish/receiver?
Thank you for the clarification in advance,
Gilbert Sebenste,
Data guru, AllisonHouse, LLC
On Tue, Jun 4, 2019 at 12:51 PM Unidata User Support <
support@xxxxxxxxxxxxxxxx> wrote:
> IMPORTANT notice of major changes to the IDD NIMAGE datastream; please
> read.
>
> First, we apologies for cross posting this announcement to multiple lists,
> but we want to make sure that all users that will/could be affected
> are notified.
>
> Second, we realize that the following message is long, but please read
> it all since it contains important information that will affect sites
> that are or will be receiving imagery products in the IDD NIMAGE feed.
>
> What:
>
> We will be transitioning the IDD NIMAGE datastream to one that
> contains all of the GOES-16 and GOES-17 Level 2 image products
> that are currently being being sent in the IDD NOTHER and HDS
> datastreams.
>
> When:
>
> It is our desire to begin distributing a reconstituted NIMAGE feed
> from our top level IDD relay clusters (idd.unidata.ucar.edu and
> iddb.unidata.ucar.edu) in the mid-June time frame. We will postpone
> turning on this feed if/when end-users indicate that they need more
> time to prepare for the change.
>
> Impact:
>
> The volume of data available in the NIMAGE feed will increase from
> an average of about 100 MB/hr to an average that is close to 7 GB/hr,
> a 70-fold increase!
>
> Reason for the change:
>
> NOAA has indicated that operation of GOES-15 will likely cease in
> early-mid July:
>
> https://www.goes-r.gov/users/transitionToOperations17.html
>
> Since GOES-East imagery in GINI format was removed from the NOAAPort
> SBN after GOES-13 was replaced as GOES-East by GOES-16 and then
> decommissioned, we believe that it is most likely that all GINI imagery
> will cease to be distributed in NOAAPort in the mid-July time frame.
> Since the NIMAGE IDD data feed has been populated solely by the GINI
> imagery distributed in NOAAPort, this feed will cease to contain any data .
>
> Currently, NOAAPort contains GOES-16 and GOES-17 ABI (Advanced Baseline
> Imager) Level 2 product tiles (pieces of full image scenes), and those
> tiles are being distributed in the IDD NOTHER data feed. To date, these
> image tiles would need to be stitched together by end-users before they
> were fully usable in display and analysis applications available from
> Unidata (e.g., AWIPS, GEMPAK, IDV, McIDAS and MetPy (this is not
> strictly true for AWIPS)).
>
> In addition to the ABI Level 2 product tiles, 19 other Level 2 products
> are also being distributed in NOAAPort, and we have been making those
> products available in the IDD NOTHER and HDS data feeds since their
> addition to NOAAPort. Like the ABI Level 2 product tiles, the other
> Level 2 products need some processing before they are usable in end-user
> applications - they need to have the NOAAPort header and footer stripped
> off to leave the underlying netCDF4 files.
>
> We believe that it will be much easier for end-users, who get their
> NOAAPort-delivered Level 2 images and products via the IDD, if we
> reconstitute full images scenes from ABI tiles and strip NOAAPort
> broadcast headers and footers here in Unidata and then distributed the
> resulting netCDF4 images/products in a reconstituted NIMAGE IDD data
> feed. Also, reconstituting the NIMAGE feed will allow us to make other,
> value-added GOES-16/17 products available to the community. The first
> of these products is GLM imagery/grids being produced at Texas Tech by
> community member Eric Bruning.
>
> Information on the imagery and products that are currently contained
> in and those that will be added to the reconstituted NIMAGE feed can be
> found in:
>
> https://www.unidata.ucar.edu/data/nimage.html
>
> We have been using all of the products in this new incarnation of the
> NIMAGE feed internally for the past few months to test their viability
> in the various application packages that we provide and support. We are
> satisfied that transition to use of these products by end users will be
> straightforward, but it will require that sites install and transition
> to use of the latest of each package that we make available.
>
> Important things to note:
>
> - the reconstitution of the NIMAGE feed comes at a cost
>
> The volume of data available will increase from an average of about
> 100 MB/hr to an average nearing 7 GB/hr, a 70-fold increase!
>
> Sites that do not want to receive any/all of the processed GOES-16/17
> imagery/products in the new NIMAGE feed need to make sure that their
> LDM configuration file REQUEST(s) do not include products that they
> do not want.
>
> The current GOES-15 GINI imagery that is being distributed in NOAAPort
> all have LDM/IDD Product IDs that begin with 'satz'. Sites requesting
> all of the current NIMAGE feed via a REQUEST that looks like:
>
> REQUEST NIMAGE ".*" idd.unidata.ucar.edu
>
> and that either do not want or can not handle the increased data volume
> should immediately change their REQUEST to something like:
>
> REQUEST NIMAGE "satz" idd.unidata.ucar.edu
>
> Sites that do want to receive all of the imagery that will be available
> in the reconstituted NIMAGE feed and whose network connection can handle
> the increased volume can leave their REQUEST(s) as is, but this decision
> should not be taken lightly given the 70-fold increase in data volume!
>
> Sites wanting to receive some of the data from the reconstituted
> NIMAGE feed are advised to read through the NIMAGE web page listed
> above to become familiar with how the LDM/IDD Product IDs will look
> and then re-fashion their feed REQUEST(s) to get the set of products
> desired.
>
> - sites that want to use the products that will be available in the
> reconstituted NIMAGE feed should immediately install and switch to
> using the most recent releases of their Unidata-supplied display and
> analysis applications
>
> This is especially true for GEMPAK users as imagery in netCDF4 format
> was not supported in Unidata GEMPAK prior to the current release.
>
> - the data volume that will be available in the reconstituted NIMAGE
> feed is actually 20% less than the volume of the sum of the products
> being delivered in NOAAPort and distributed in the NOTHER and HDS
> datastreams
>
> The reason for this is twofold:
>
> - NOAAPort broadcast headers and footers have been removed
> resulting in a minor reduction of volume
>
> - netCDF4 compression has been implemented/implemented better in
> the products
>
> Recap:
>
> The reconstituted IDD NIMAGE datastream will contain:
>
> - full image scenes that have been created by stitching together
> the ABI tiles sent in the NOAAPort SBN and distributed in the
> IDD NOTHER feed
>
> - all other Level 2 products being distributed in the NOAAPort SBN
> in the IDD NOTHER and HDS feeds
>
> - value added GLM image/grid products created by Eric Bruning
> of Texas Tech University
>
> - all products are in netCDF-4 format
>
> - all products are in the GOES-R mission standard Fixed Grid
> projection
>
> - the names for all products (contained in the LDM/IDD Product IDs)
> follow the GOES-R mission standard naming convention as detailed
> in:
>
> GOES-R SERIES PRODUCT DEFINITION AND USERS’ GUIDE (PUG)
> https://www.goes-r.gov/users/docs/PUG-GRB-vol4.pdf
>
> In all cases, the LDM/IDD Product IDs will consist of fully qualified
> path names for products, and the file name portion of the fully
> qualified names follow GOES-R mission standards for naming. The NIMAGE
> web page referenced above lists example Product IDs for each product.
>
> Please send any/all questions/comments about the upcoming change to the
> IDD NIMAGE feed to:
>
> support-datastream@xxxxxxxxxxxxxxxx
>
> Best regards,
>
> Unidata User Support
> --
> ********************************************************************** <
> Unidata User Support UCAR Unidata Program <
> (303)497-8643 P.O. Box 3000 <
> support@xxxxxxxxxxxxxxxx Boulder, CO 80307 <
> ---------------------------------------------------------------------- <
> Unidata Web Support http://www.unidata.ucar.edu/support <
> ********************************************************************** <
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
--
----
Gilbert Sebenste
Consulting Meteorologist
AllisonHouse, LLC
