<html><head></head><body><img src="cid:ncoEmailLogo.png@SNC.f19f4f18a72fae15"
height="50"> <b>INC0316836 has new comments ( GOES
Imagery)(Active)</b><hr>
<style>.hideInEmail { display: none; }; </style>
<div></div>
<div><p>Some details of the ticket are included below:</p><table><tbody
style="vertical-align:top;"><tr><th
style="min-width:150px;"></th><th></th></tr><tr><td><b>End
User:</b></td><td>NCO Ops</td></tr><tr><td><b>Short description:</b></td><td>
GOES Imagery</td></tr><tr><td><b>Customer Impact Statement:</b></td><td>missing
tiles in GOES
imagery</td></tr><tr><td><b>State:</b></td><td>Active</td></tr><tr><td><b>Priority:</b></td><td>High</td></tr><tr><td><b>Category:</b></td><td>General</td></tr><tr><td><b>Sub-Category:</b></td><td>Other</td></tr><tr><td><b>Assignment
group:</b></td><td>NCO Help Desk
Triage</td></tr><tr><td><b>Description:</b></td><td>Received email notification
from Jeff Kelley: Hi NCO. Have you been in contact with NESDIS at all about
the missing tiles in GOES-16? Every image that comes in has missing tiles,
whether in AWIPS, the FAA weather system, or even the College of Dupage
website.
<br><br>https://weather.cod.edu/satrad/?parms=continental-conus-13-24-1-100-1&checked=map&colorbar=undefined<br><br>I
haven't seen any ADA or ADM messages either.<br><br>Jeff<br>ZID
CWSU</td></tr></tbody></table></div>
<div><p>The following comments were given about the Incident:</p><div
style="margin-left:30px;"><table style="width:100%;"><tbody><tr><td><div
style="padding:10px 15px; border:1px solid lightgray; border-left:7px solid
Purple;margin:2px 0;"><strong>2021-12-13 20:14:24 EST - Guest </strong><sup
style="float:right;">Additional comments</sup><div><span
style="word-wrap:break-word;display:block;">reply from:
gilbert@xxxxxxxxxxxxxxxx<br><br>Someone from the College of DuPage reached out
to me with an interesting observation from the University of
Oklahoma...<br><br>We had an outage on our NOAAport receive system this
morning, which caused our systems to fail over to our partner organization's
feeds. On investigation, we found that our receiver, ingest software, and LDM
software were all working fine. We saw packets coming into the Linux-based
ingest server, but nothing was being ingested. Then, it was noticed that the
source IP of the NOAAport multicast traffic had changed, for at least some of
the packets. Whereas in the past, it had been 10.0.9.51 , we are now seeing
packets from 10.0.9.52 . Our ingest server was not accepting those because it
had no special routing rule for that IP. We added such a rule (and did *not*
delete the existing one for the .51 IP) and our NOAAport ingest began working
as before.<br><br>I can confirm the source IP for me is 10.0.9.52 as well, but
unlike him that IP range was already open on my system and that traffic was
making its way in. I'm not sure if this is related or not, but I wanted to
pass along the intel in case it is.</span></div></div></td></tr><tr><td><div
style="padding:10px 15px; border:1px solid lightgray; border-left:7px solid
Purple;margin:2px 0;"><strong>2021-12-13 19:54:36 EST - Jeff
Kelley</strong><sup style="float:right;">Additional comments</sup><div><span
style="word-wrap:break-word;display:block;">reply from:
jeff.kelley@xxxxxxxx<br><br>I just got off the phone with NCF. They are aware
that data dropouts are<br>occurring (with all data types, not just satellite)
after switching between<br>systems earlier today. The tech mentioned that it
might be a
firewall<br>issue.<br><br>Jeff</span></div></div></td></tr><tr><td><div
style="padding:10px 15px; border:1px solid lightgray; border-left:7px solid
Purple;margin:2px 0;"><strong>2021-12-13 19:53:01 EST - Shellene
Cope</strong><sup style="float:right;">Additional comments</sup><div><span
style="word-wrap:break-word;display:block;">Gilbert Sebenste emailed: 1) I
have been using the GOES Restitch for a couple of months now and it has worked
perfectly until today. Today I am getting partial images. Upon looking at the
log files I am finding that it is "Timing Out" waiting for the tile to arrive.
I am not seeing any large amounts of dropped packets. I am not sure what could
be causing my sudden issue. Any insight as to what I might check?
<br>
<br>
<br>
<br>Thanks, Jim ( partial log below)
<br>
<br>
<br>
<br>
<br>
<br>[TIRE 132051] 2021-12-13 20:54:52,378 [process_items]:
OR_ABI-L2-CMIPC-M6C05_G16_s20213472051170_e20213472051170_c20213472051170.nc: 8
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:54:53,177 [process_items]:
OR_ABI-L2-CMIPC-M6C01_G16_s20213472051170_e20213472051170_c20213472051170.nc: 7
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:54:54,033 [process_items]:
OR_ABI-L2-CMIPC-M6C01_G16_s20213472051170_e20213472051170_c20213472051170.nc: 8
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:00,091 [process_items]:
OR_ABI-L2-CMIPC-M6C03_G16_s20213472051170_e20213472051170_c20213472051170.nc:
11 of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:00,753 [process_items]:
OR_ABI-L2-CMIPC-M6C01_G16_s20213472051170_e20213472051170_c20213472051170.nc: 9
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:01,554 [process_items]:
OR_ABI-L2-CMIPC-M6C03_G16_s20213472051170_e20213472051170_c20213472051170.nc:
12 of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:06,066 [process_items]:
OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 6
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:06,350 [process_items]:
OR_ABI-L2-CMIPC-M6C03_G16_s20213472051170_e20213472051170_c20213472051170.nc:
13 of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:07,089 [process_items]:
OR_ABI-L2-CMIPC-M6C05_G16_s20213472051170_e20213472051170_c20213472051170.nc: 9
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:07,592 [process_items]:
OR_ABI-L2-CMIPC-M6C01_G16_s20213472051170_e20213472051170_c20213472051170.nc:
10 of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:08,514 [process_items]:
OR_ABI-L2-CMIPC-M6C05_G16_s20213472051170_e20213472051170_c20213472051170.nc:
10 of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:13,149 [process_items]:
OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 7
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:16,471 [process_items]:
OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 8
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:22,188 [process_items]:
OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 9
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:34,664 [process_items]:
OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc:
10 of 15 tiles received.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:44,834 [process_items]: Finishing due to
timeout.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:44,944 [store_done]:
GOES16/Products/CloudAndMoistureImagery/CONUS/Channel07/20211213/OR_ABI-L2-CMIPC-M6C07_G16_s20213472046170_e20213472046170_c20213472046170.nc
finished.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:44,947 [process_items]: Finishing due to
timeout.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:45,051 [store_done]:
GOES16/Products/CloudAndMoistureImagery/CONUS/Channel12/20211213/OR_ABI-L2-CMIPC-M6C12_G16_s20213472046170_e20213472046170_c20213472046170.nc
finished.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:45,698 [process_items]:
OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc:
11 of 15 tiles received.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:47,772 [process_items]: Finishing due to
timeout.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:47,875 [store_done]:
GOES16/Products/CloudAndMoistureImagery/CONUS/Channel10/20211213/OR_ABI-L2-CMIPC-M6C10_G16_s20213472046170_e20213472046170_c20213472046170.nc
finished.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:48,075 [process_items]: Finishing due to
timeout.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:48,181 [store_done]:
GOES16/Products/CloudAndMoistureImagery/CONUS/Channel11/20211213/OR_ABI-L2-CMIPC-M6C11_G16_s20213472046170_e20213472046170_c20213472046170.nc
finished.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:49,455 [process_items]: Finishing due to
timeout.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:49,559 [store_done]:
GOES16/Products/CloudAndMoistureImagery/CONUS/Channel09/20211213/OR_ABI-L2-CMIPC-M6C09_G16_s20213472046170_e20213472046170_c20213472046170.nc
finished.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:49,560 [process_items]: Finishing due to
timeout.
<br>
<br>
<br>
<br>[TIRE 132046] 2021-12-13 20:55:49,664 [store_done]:
GOES16/Products/CloudAndMoistureImagery/CONUS/Channel08/20211213/OR_ABI-L2-CMIPC-M6C08_G16_s20213472046170_e20213472046170_c20213472046170.nc
finished.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:49,891 [process_items]:
OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc:
12 of 15 tiles received.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:51,447 [process_items]: Finishing due to
timeout.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:51,556 [store_done]:
GOES16/Products/CloudAndMoistureImagery/CONUS/Channel13/20211213/OR_ABI-L2-CMIPC-M6C13_G16_s20213472046170_e20213472046170_c20213472046170.nc
finished.
<br>
<br>2) Hi Jim,
<br>
<br>I was just about to write my own email about this. It seems to me that not
all GOES-16 tiles are making it to NOAAPort. I was about to ask if anyone else
is seeing this, and I guess you are.
<br>
<br>I noticed this began around 14Z today. GOES-16 CONUS & Full-Disk are
impacted, Mesoscale and GOES-17 seem okay. I haven't seen any announcements
about data flow issues. Using notifyme I saw that Unidata was missing the same
tiles I was.
<br>
<br>Does anyone else have any insight?
<br>
<br>-Mike
<br>
<br>3) Hello NCO,
<br>
<br>Some GOES-16 tiles on NOAAport/SBN are missing. This started just after 14Z,
<br>when you awitched to the BMGS. See the thread below from a UNIDATA user log.
<br>Multiple people are seeing this.
<br>
<br>Gilbert Sebenste
<br>Meteorological Consultant
<br>AllisonHouse, LLC
<br>
<br>(SEC)</span></div></div></td></tr><tr><td><div style="padding:10px 15px;
border:1px solid lightgray; border-left:7px solid Purple;margin:2px
0;"><strong>2021-12-13 19:48:34 EST - Shellene Cope</strong><sup
style="float:right;">Additional comments</sup><div><span
style="word-wrap:break-word;display:block;">Jeff Kelley has emailed the
following reply: There has been no improvement in this issue. Have you heard
anything from NESDIS? It's strange that the imagery on the STAR and NASA
websites appear fine.
<br>
<br>https://weather.msfc.nasa.gov/cgi-bin/basicLooper.pl?category=goeseastabiconus&regex=10p35um&title=GOES-East%20ABI%20CONUS%20-%20%2010.35%20um%20-%20Band%2013&time_drop=show
<br>https://www.star.nesdis.noaa.gov/GOES/conus_band.php?sat=G16&band=13&length=24
<br>
<br>AWIPS and the FAA weather system continue to exhibit the same issue as this
link: https://weather.cod.edu/satrad/
<br>
<br>(SEC) </span></div></div></td></tr></tbody></table></div></div>
<div><p>The following work notes were given about the Incident:</p><div
style="margin-left:30px;"><table style="width:100%;"><tbody><tr><td><div
style="padding:10px 15px; border:1px solid lightgray; border-left:7px solid
gold;margin:2px 0;"><strong>2021-12-13 20:10:02 EST - Curtis Sims</strong><sup
style="float:right;">Work notes</sup><div><span
style="word-wrap:break-word;display:block;">@0054z - OMB received an update
from Jeff Kelley
<br>reply from: jeff.kelley@xxxxxxxx
<br>
<br>I just got off the phone with NCF. They are aware that data dropouts are
<br>occurring (with all data types, not just satellite) after switching between
<br>systems earlier today. The tech mentioned that it might be a firewall
<br>issue.
<br>
<br>Jeff
<br>(CS)</span></div></div></td></tr></tbody></table></div></div>
<hr>
<div> </div><div style="display:inline">Ref:MSG5533548</div></body></html>
