<html><head></head><body><img src="cid:ncoEmailLogo.png@SNC.a8e002813250a116"
height="50"> <b>INC0316836 has new comments ( GOES
Imagery)(Active)</b><hr>
<style>.hideInEmail { display: none; }; </style>
<div></div>
<div><p>Some details of the ticket are included below:</p><table><tbody
style="vertical-align:top;"><tr><th
style="min-width:150px;"></th><th></th></tr><tr><td><b>End
User:</b></td><td>Jeff Kelley</td></tr><tr><td><b>Short
description:</b></td><td> GOES Imagery</td></tr><tr><td><b>Customer Impact
Statement:</b></td><td>missing tiles in GOES
imagery</td></tr><tr><td><b>State:</b></td><td>Active</td></tr><tr><td><b>Priority:</b></td><td>High</td></tr><tr><td><b>Queue
position:</b></td><td>152</td></tr><tr><td><b>Category:</b></td><td>General</td></tr><tr><td><b>Sub-Category:</b></td><td>Other</td></tr><tr><td><b>Assignment
group:</b></td><td>NCO OMB Tech
Control</td></tr><tr><td><b>Description:</b></td><td>Received email
notification from Jeff Kelley: Hi NCO. Have you been in contact with NESDIS at
all about the missing tiles in GOES-16? Every image that comes in has missing
tiles, whether in AWIPS, the FAA weather system, or even the College of Dupage
website.
<br><br>https://weather.cod.edu/satrad/?parms=continental-conus-13-24-1-100-1&checked=map&colorbar=undefined<br><br>I
haven't seen any ADA or ADM messages either.<br><br>Jeff<br>ZID
CWSU</td></tr></tbody></table></div>
<div><p>The following comments were given about the Incident:</p><div
style="margin-left:30px;"><table style="width:100%;"><tbody><tr><td><div
style="padding:10px 15px; border:1px solid lightgray; border-left:7px solid
Purple;margin:2px 0;"><strong>2021-12-16 10:22:47 EST - Guest </strong><sup
style="float:right;">Additional comments</sup><div><span
style="word-wrap:break-word;display:block;">reply from:
gilbert@xxxxxxxxxxxxxxxx<br><br>This issue has started again as of 14:30Z this
morning.</span></div></div></td></tr><tr><td><div style="padding:10px 15px;
border:1px solid lightgray; border-left:7px solid Purple;margin:2px
0;"><strong>2021-12-13 20:14:24 EST - Guest </strong><sup
style="float:right;">Additional comments</sup><div><span
style="word-wrap:break-word;display:block;">reply from:
gilbert@xxxxxxxxxxxxxxxx<br><br>Someone from the College of DuPage reached out
to me with an interesting observation from the University of
Oklahoma...<br><br>We had an outage on our NOAAport receive system this
morning, which caused our systems to fail over to our partner organization's
feeds. On investigation, we found that our receiver, ingest software, and LDM
software were all working fine. We saw packets coming into the Linux-based
ingest server, but nothing was being ingested. Then, it was noticed that the
source IP of the NOAAport multicast traffic had changed, for at least some of
the packets. Whereas in the past, it had been 10.0.9.51 , we are now seeing
packets from 10.0.9.52 . Our ingest server was not accepting those because it
had no special routing rule for that IP. We added such a rule (and did *not*
delete the existing one for the .51 IP) and our NOAAport ingest began working
as before.<br><br>I can confirm the source IP for me is 10.0.9.52 as well, but
unlike him that IP range was already open on my system and that traffic was
making its way in. I'm not sure if this is related or not, but I wanted to
pass along the intel in case it is.</span></div></div></td></tr><tr><td><div
style="padding:10px 15px; border:1px solid lightgray; border-left:7px solid
Purple;margin:2px 0;"><strong>2021-12-13 19:54:36 EST - Jeff
Kelley</strong><sup style="float:right;">Additional comments</sup><div><span
style="word-wrap:break-word;display:block;">reply from:
jeff.kelley@xxxxxxxx<br><br>I just got off the phone with NCF. They are aware
that data dropouts are<br>occurring (with all data types, not just satellite)
after switching between<br>systems earlier today. The tech mentioned that it
might be a
firewall<br>issue.<br><br>Jeff</span></div></div></td></tr><tr><td><div
style="padding:10px 15px; border:1px solid lightgray; border-left:7px solid
Purple;margin:2px 0;"><strong>2021-12-13 19:53:01 EST - Shellene
Cope</strong><sup style="float:right;">Additional comments</sup><div><span
style="word-wrap:break-word;display:block;">Gilbert Sebenste emailed: 1) I
have been using the GOES Restitch for a couple of months now and it has worked
perfectly until today. Today I am getting partial images. Upon looking at the
log files I am finding that it is "Timing Out" waiting for the tile to arrive.
I am not seeing any large amounts of dropped packets. I am not sure what could
be causing my sudden issue. Any insight as to what I might check?
<br>
<br>
<br>
<br>Thanks, Jim ( partial log below)
<br>
<br>
<br>
<br>
<br>
<br>[TIRE 132051] 2021-12-13 20:54:52,378 [process_items]:
OR_ABI-L2-CMIPC-M6C05_G16_s20213472051170_e20213472051170_c20213472051170.nc: 8
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:54:53,177 [process_items]:
OR_ABI-L2-CMIPC-M6C01_G16_s20213472051170_e20213472051170_c20213472051170.nc: 7
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:54:54,033 [process_items]:
OR_ABI-L2-CMIPC-M6C01_G16_s20213472051170_e20213472051170_c20213472051170.nc: 8
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:00,091 [process_items]:
OR_ABI-L2-CMIPC-M6C03_G16_s20213472051170_e20213472051170_c20213472051170.nc:
11 of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:00,753 [process_items]:
OR_ABI-L2-CMIPC-M6C01_G16_s20213472051170_e20213472051170_c20213472051170.nc: 9
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:01,554 [process_items]:
OR_ABI-L2-CMIPC-M6C03_G16_s20213472051170_e20213472051170_c20213472051170.nc:
12 of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:06,066 [process_items]:
OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 6
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:06,350 [process_items]:
OR_ABI-L2-CMIPC-M6C03_G16_s20213472051170_e20213472051170_c20213472051170.nc:
13 of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:07,089 [process_items]:
OR_ABI-L2-CMIPC-M6C05_G16_s20213472051170_e20213472051170_c20213472051170.nc: 9
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:07,592 [process_items]:
OR_ABI-L2-CMIPC-M6C01_G16_s20213472051170_e20213472051170_c20213472051170.nc:
10 of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:08,514 [process_items]:
OR_ABI-L2-CMIPC-M6C05_G16_s20213472051170_e20213472051170_c20213472051170.nc:
10 of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:13,149 [process_items]:
OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 7
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:16,471 [process_items]:
OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 8
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:22,188 [process_items]:
OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 9
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:34,664 [process_items]:
OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc:
10 of 15 tiles received.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:44,834 [process_items]: Finishing due to
timeout.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:44,944 [store_done]:
GOES16/Products/CloudAndMoistureImagery/CONUS/Channel07/20211213/OR_ABI-L2-CMIPC-M6C07_G16_s20213472046170_e20213472046170_c20213472046170.nc
finished.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:44,947 [process_items]: Finishing due to
timeout.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:45,051 [store_done]:
GOES16/Products/CloudAndMoistureImagery/CONUS/Channel12/20211213/OR_ABI-L2-CMIPC-M6C12_G16_s20213472046170_e20213472046170_c20213472046170.nc
finished.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:45,698 [process_items]:
OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc:
11 of 15 tiles received.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:47,772 [process_items]: Finishing due to
timeout.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:47,875 [store_done]:
GOES16/Products/CloudAndMoistureImagery/CONUS/Channel10/20211213/OR_ABI-L2-CMIPC-M6C10_G16_s20213472046170_e20213472046170_c20213472046170.nc
finished.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:48,075 [process_items]: Finishing due to
timeout.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:48,181 [store_done]:
GOES16/Products/CloudAndMoistureImagery/CONUS/Channel11/20211213/OR_ABI-L2-CMIPC-M6C11_G16_s20213472046170_e20213472046170_c20213472046170.nc
finished.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:49,455 [process_items]: Finishing due to
timeout.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:49,559 [store_done]:
GOES16/Products/CloudAndMoistureImagery/CONUS/Channel09/20211213/OR_ABI-L2-CMIPC-M6C09_G16_s20213472046170_e20213472046170_c20213472046170.nc
finished.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:49,560 [process_items]: Finishing due to
timeout.
<br>
<br>
<br>
<br>[TIRE 132046] 2021-12-13 20:55:49,664 [store_done]:
GOES16/Products/CloudAndMoistureImagery/CONUS/Channel08/20211213/OR_ABI-L2-CMIPC-M6C08_G16_s20213472046170_e20213472046170_c20213472046170.nc
finished.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:49,891 [process_items]:
OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc:
12 of 15 tiles received.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:51,447 [process_items]: Finishing due to
timeout.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:51,556 [store_done]:
GOES16/Products/CloudAndMoistureImagery/CONUS/Channel13/20211213/OR_ABI-L2-CMIPC-M6C13_G16_s20213472046170_e20213472046170_c20213472046170.nc
finished.
<br>
<br>2) Hi Jim,
<br>
<br>I was just about to write my own email about this. It seems to me that not
all GOES-16 tiles are making it to NOAAPort. I was about to ask if anyone else
is seeing this, and I guess you are.
<br>
<br>I noticed this began around 14Z today. GOES-16 CONUS & Full-Disk are
impacted, Mesoscale and GOES-17 seem okay. I haven't seen any announcements
about data flow issues. Using notifyme I saw that Unidata was missing the same
tiles I was.
<br>
<br>Does anyone else have any insight?
<br>
<br>-Mike
<br>
<br>3) Hello NCO,
<br>
<br>Some GOES-16 tiles on NOAAport/SBN are missing. This started just after 14Z,
<br>when you awitched to the BMGS. See the thread below from a UNIDATA user log.
<br>Multiple people are seeing this.
<br>
<br>Gilbert Sebenste
<br>Meteorological Consultant
<br>AllisonHouse, LLC
<br>
<br>(SEC)</span></div></div></td></tr><tr><td><div style="padding:10px 15px;
border:1px solid lightgray; border-left:7px solid Purple;margin:2px
0;"><strong>2021-12-13 19:48:34 EST - Shellene Cope</strong><sup
style="float:right;">Additional comments</sup><div><span
style="word-wrap:break-word;display:block;">Jeff Kelley has emailed the
following reply: There has been no improvement in this issue. Have you heard
anything from NESDIS? It's strange that the imagery on the STAR and NASA
websites appear fine.
<br>
<br>https://weather.msfc.nasa.gov/cgi-bin/basicLooper.pl?category=goeseastabiconus&regex=10p35um&title=GOES-East%20ABI%20CONUS%20-%20%2010.35%20um%20-%20Band%2013&time_drop=show
<br>https://www.star.nesdis.noaa.gov/GOES/conus_band.php?sat=G16&band=13&length=24
<br>
<br>AWIPS and the FAA weather system continue to exhibit the same issue as this
link: https://weather.cod.edu/satrad/
<br>
<br>(SEC) </span></div></div></td></tr></tbody></table></div></div>
<div><p>The following work notes were given about the Incident:</p><div
style="margin-left:30px;"><table style="width:100%;"><tbody><tr><td><div
style="padding:10px 15px; border:1px solid lightgray; border-left:7px solid
gold;margin:2px 0;"><strong>2021-12-13 20:39:36 EST - Curtis Sims</strong><sup
style="float:right;">Work notes</sup><div><span
style="word-wrap:break-word;display:block;">@0114z - OMB received an update
from gilbert@xxxxxxxxxxxxxxxx
<br>reply from: gilbert@xxxxxxxxxxxxxxxx
<br>
<br>Someone from the College of DuPage reached out to me with an interesting
observation from the University of Oklahoma...
<br>
<br>We had an outage on our NOAAport receive system this morning, which caused
our systems to fail over to our partner organization's feeds. On investigation,
we found that our receiver, ingest software, and LDM software were all working
fine. We saw packets coming into the Linux-based ingest server, but nothing was
being ingested. Then, it was noticed that the source IP of the NOAAport
multicast traffic had changed, for at least some of the packets. Whereas in the
past, it had been 10.0.9.51 , we are now seeing packets from 10.0.9.52 . Our
ingest server was not accepting those because it had no special routing rule
for that IP. We added such a rule (and did *not* delete the existing one for
the .51 IP) and our NOAAport ingest began working as before.
<br>
<br>I can confirm the source IP for me is 10.0.9.52 as well, but unlike him
that IP range was already open on my system and that traffic was making its way
in. I'm not sure if this is related or not, but I wanted to pass along the
intel in case it is.
<br>(CS)</span></div></div></td></tr><tr><td><div style="padding:10px 15px;
border:1px solid lightgray; border-left:7px solid gold;margin:2px
0;"><strong>2021-12-13 20:10:02 EST - Curtis Sims</strong><sup
style="float:right;">Work notes</sup><div><span
style="word-wrap:break-word;display:block;">@0054z - OMB received an update
from Jeff Kelley
<br>reply from: jeff.kelley@xxxxxxxx
<br>
<br>I just got off the phone with NCF. They are aware that data dropouts are
<br>occurring (with all data types, not just satellite) after switching between
<br>systems earlier today. The tech mentioned that it might be a firewall
<br>issue.
<br>
<br>Jeff
<br>(CS)</span></div></div></td></tr></tbody></table></div></div>
<hr>
<div> </div><div style="display:inline">Ref:MSG5547071</div></body></html>
