<html><head></head><body><img src="cid:ncoEmailLogo.png@SNC.aaee285474011ea5"
height="50"> <b>INC0316836 has been Resolved (
GOES Imagery)</b><hr>
<style>.hideInEmail { display: none; }; </style>
<div></div>
<div><p>Some details of the ticket are included below:</p><table><tbody
style="vertical-align:top;"><tr><th
style="min-width:150px;"></th><th></th></tr><tr><td><b>End
User:</b></td><td>Jeff Kelley</td></tr><tr><td><b>Short
description:</b></td><td> GOES Imagery</td></tr><tr><td><b>Customer Impact
Statement:</b></td><td>missing tiles in GOES
imagery</td></tr><tr><td><b>State:</b></td><td>Resolved</td></tr><tr><td><b>Priority:</b></td><td>High</td></tr><tr><td><b>Category:</b></td><td>General</td></tr><tr><td><b>Sub-Category:</b></td><td>Other</td></tr><tr><td><b>Assigned
to:</b></td><td>Chun Lam</td></tr><tr><td><b>Assignment group:</b></td><td>NCO
OMB Tech Control</td></tr><tr><td><b>Description:</b></td><td>Received email
notification from Jeff Kelley: Hi NCO. Have you been in contact with NESDIS at
all about the missing tiles in GOES-16? Every image that comes in has missing
tiles, whether in AWIPS, the FAA weather system, or even the College of Dupage
website.
<br><br>https://weather.cod.edu/satrad/?parms=continental-conus-13-24-1-100-1&checked=map&colorbar=undefined<br><br>I
haven't seen any ADA or ADM messages either.<br><br>Jeff<br>ZID
CWSU</td></tr><tr><td><b>Close notes:</b></td><td>[NCO-MCL] Guest reported the
following:<br><br>Additional comments•03-26-2022 23:43:34just now<br>reply
from: gilbert@xxxxxxxxxxxxxxxx<br><br>This issue has been solved on our end. It
was corrected several weeks later.<br>My bad for not mailing back and
requesting that the ticket be closed. My apolgies, and I deem this issue to be
solved.<br><br>Gilbert Sebenste<br>AllisonHouse,
LLC<br></td></tr></tbody></table></div>
<div><p>The following comments were given about the Incident:</p><div
style="margin-left:30px;"><table style="width:100%;"><tbody><tr><td><div
style="padding:10px 15px; border:1px solid lightgray; border-left:7px solid
Purple;margin:2px 0;"><strong>2022-03-26 19:44:51 EDT - Chun Lam</strong><sup
style="float:right;">Additional comments</sup><div><span
style="word-wrap:break-word;display:block;"> Closed Code: Closed/Resolved by
Caller<br>Closed Notes: [NCO-MCL] Guest reported the following:
<br>
<br>Additional comments•03-26-2022 23:43:34just now
<br>reply from: gilbert@xxxxxxxxxxxxxxxx
<br>
<br>This issue has been solved on our end. It was corrected several weeks later.
<br>My bad for not mailing back and requesting that the ticket be closed. My
apolgies, and I deem this issue to be solved.
<br>
<br>Gilbert Sebenste
<br>AllisonHouse, LLC
<br></span></div></div></td></tr><tr><td><div style="padding:10px 15px;
border:1px solid lightgray; border-left:7px solid Purple;margin:2px
0;"><strong>2022-03-26 19:43:34 EDT - Guest </strong><sup
style="float:right;">Additional comments</sup><div><span
style="word-wrap:break-word;display:block;">reply from:
gilbert@xxxxxxxxxxxxxxxx<br><br>This issue has been solved on our end. It was
corrected several weeks later.<br>My bad for not mailing back and requesting
that the ticket be closed. My apolgies, and I deem this issue to be
solved.<br><br>Gilbert Sebenste<br>AllisonHouse,
LLC</span></div></div></td></tr><tr><td><div style="padding:10px 15px;
border:1px solid lightgray; border-left:7px solid Purple;margin:2px
0;"><strong>2022-03-26 19:34:27 EDT - Chun Lam</strong><sup
style="float:right;">Additional comments</sup><div><span
style="word-wrap:break-word;display:block;">[NCO-MCL] NCO is requesting via
email an update from Jeff Kelley to see if the issue still
exists.</span></div></div></td></tr><tr><td><div style="padding:10px 15px;
border:1px solid lightgray; border-left:7px solid Purple;margin:2px
0;"><strong>2021-12-16 10:52:44 EST - Jeff Kelley</strong><sup
style="float:right;">Additional comments</sup><div><span
style="word-wrap:break-word;display:block;">reply from:
jeff.kelley@xxxxxxxx<br><br>Seeing the same thing at the CWSU in Indianapolis,
in AWIPS and the FAA<br>weather
system.<br><br>Jeff</span></div></div></td></tr><tr><td><div
style="padding:10px 15px; border:1px solid lightgray; border-left:7px solid
Purple;margin:2px 0;"><strong>2021-12-16 10:40:49 EST - Guest </strong><sup
style="float:right;">Additional comments</sup><div><span
style="word-wrap:break-word;display:block;">reply from:
gilbert@xxxxxxxxxxxxxxxx<br><br>NCO,<br><br>You can see this issue by running
"notifyme -vl- -h localhost -f NOTHER -p TIRE02". That will look for all
GOES-East CONUS Band 02 tiles, they'll have an identifier at the end of PAx
where x will be an increasing letter representing each tile. There should be
15 tiles for that product, and quite a few aren't being
seen.</span></div></div></td></tr><tr><td><div style="padding:10px 15px;
border:1px solid lightgray; border-left:7px solid Purple;margin:2px
0;"><strong>2021-12-16 10:22:47 EST - Guest </strong><sup
style="float:right;">Additional comments</sup><div><span
style="word-wrap:break-word;display:block;">reply from:
gilbert@xxxxxxxxxxxxxxxx<br><br>This issue has started again as of 14:30Z this
morning.</span></div></div></td></tr><tr><td><div style="padding:10px 15px;
border:1px solid lightgray; border-left:7px solid Purple;margin:2px
0;"><strong>2021-12-13 20:14:24 EST - Guest </strong><sup
style="float:right;">Additional comments</sup><div><span
style="word-wrap:break-word;display:block;">reply from:
gilbert@xxxxxxxxxxxxxxxx<br><br>Someone from the College of DuPage reached out
to me with an interesting observation from the University of
Oklahoma...<br><br>We had an outage on our NOAAport receive system this
morning, which caused our systems to fail over to our partner organization's
feeds. On investigation, we found that our receiver, ingest software, and LDM
software were all working fine. We saw packets coming into the Linux-based
ingest server, but nothing was being ingested. Then, it was noticed that the
source IP of the NOAAport multicast traffic had changed, for at least some of
the packets. Whereas in the past, it had been 10.0.9.51 , we are now seeing
packets from 10.0.9.52 . Our ingest server was not accepting those because it
had no special routing rule for that IP. We added such a rule (and did *not*
delete the existing one for the .51 IP) and our NOAAport ingest began working
as before.<br><br>I can confirm the source IP for me is 10.0.9.52 as well, but
unlike him that IP range was already open on my system and that traffic was
making its way in. I'm not sure if this is related or not, but I wanted to
pass along the intel in case it is.</span></div></div></td></tr><tr><td><div
style="padding:10px 15px; border:1px solid lightgray; border-left:7px solid
Purple;margin:2px 0;"><strong>2021-12-13 19:54:36 EST - Jeff
Kelley</strong><sup style="float:right;">Additional comments</sup><div><span
style="word-wrap:break-word;display:block;">reply from:
jeff.kelley@xxxxxxxx<br><br>I just got off the phone with NCF. They are aware
that data dropouts are<br>occurring (with all data types, not just satellite)
after switching between<br>systems earlier today. The tech mentioned that it
might be a
firewall<br>issue.<br><br>Jeff</span></div></div></td></tr><tr><td><div
style="padding:10px 15px; border:1px solid lightgray; border-left:7px solid
Purple;margin:2px 0;"><strong>2021-12-13 19:53:01 EST - Shellene
Cope</strong><sup style="float:right;">Additional comments</sup><div><span
style="word-wrap:break-word;display:block;">Gilbert Sebenste emailed: 1) I
have been using the GOES Restitch for a couple of months now and it has worked
perfectly until today. Today I am getting partial images. Upon looking at the
log files I am finding that it is "Timing Out" waiting for the tile to arrive.
I am not seeing any large amounts of dropped packets. I am not sure what could
be causing my sudden issue. Any insight as to what I might check?
<br>
<br>
<br>
<br>Thanks, Jim ( partial log below)
<br>
<br>
<br>
<br>
<br>
<br>[TIRE 132051] 2021-12-13 20:54:52,378 [process_items]:
OR_ABI-L2-CMIPC-M6C05_G16_s20213472051170_e20213472051170_c20213472051170.nc: 8
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:54:53,177 [process_items]:
OR_ABI-L2-CMIPC-M6C01_G16_s20213472051170_e20213472051170_c20213472051170.nc: 7
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:54:54,033 [process_items]:
OR_ABI-L2-CMIPC-M6C01_G16_s20213472051170_e20213472051170_c20213472051170.nc: 8
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:00,091 [process_items]:
OR_ABI-L2-CMIPC-M6C03_G16_s20213472051170_e20213472051170_c20213472051170.nc:
11 of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:00,753 [process_items]:
OR_ABI-L2-CMIPC-M6C01_G16_s20213472051170_e20213472051170_c20213472051170.nc: 9
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:01,554 [process_items]:
OR_ABI-L2-CMIPC-M6C03_G16_s20213472051170_e20213472051170_c20213472051170.nc:
12 of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:06,066 [process_items]:
OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 6
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:06,350 [process_items]:
OR_ABI-L2-CMIPC-M6C03_G16_s20213472051170_e20213472051170_c20213472051170.nc:
13 of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:07,089 [process_items]:
OR_ABI-L2-CMIPC-M6C05_G16_s20213472051170_e20213472051170_c20213472051170.nc: 9
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:07,592 [process_items]:
OR_ABI-L2-CMIPC-M6C01_G16_s20213472051170_e20213472051170_c20213472051170.nc:
10 of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:08,514 [process_items]:
OR_ABI-L2-CMIPC-M6C05_G16_s20213472051170_e20213472051170_c20213472051170.nc:
10 of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:13,149 [process_items]:
OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 7
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:16,471 [process_items]:
OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 8
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:22,188 [process_items]:
OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 9
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:34,664 [process_items]:
OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc:
10 of 15 tiles received.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:44,834 [process_items]: Finishing due to
timeout.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:44,944 [store_done]:
GOES16/Products/CloudAndMoistureImagery/CONUS/Channel07/20211213/OR_ABI-L2-CMIPC-M6C07_G16_s20213472046170_e20213472046170_c20213472046170.nc
finished.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:44,947 [process_items]: Finishing due to
timeout.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:45,051 [store_done]:
GOES16/Products/CloudAndMoistureImagery/CONUS/Channel12/20211213/OR_ABI-L2-CMIPC-M6C12_G16_s20213472046170_e20213472046170_c20213472046170.nc
finished.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:45,698 [process_items]:
OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc:
11 of 15 tiles received.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:47,772 [process_items]: Finishing due to
timeout.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:47,875 [store_done]:
GOES16/Products/CloudAndMoistureImagery/CONUS/Channel10/20211213/OR_ABI-L2-CMIPC-M6C10_G16_s20213472046170_e20213472046170_c20213472046170.nc
finished.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:48,075 [process_items]: Finishing due to
timeout.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:48,181 [store_done]:
GOES16/Products/CloudAndMoistureImagery/CONUS/Channel11/20211213/OR_ABI-L2-CMIPC-M6C11_G16_s20213472046170_e20213472046170_c20213472046170.nc
finished.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:49,455 [process_items]: Finishing due to
timeout.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:49,559 [store_done]:
GOES16/Products/CloudAndMoistureImagery/CONUS/Channel09/20211213/OR_ABI-L2-CMIPC-M6C09_G16_s20213472046170_e20213472046170_c20213472046170.nc
finished.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:49,560 [process_items]: Finishing due to
timeout.
<br>
<br>
<br>
<br>[TIRE 132046] 2021-12-13 20:55:49,664 [store_done]:
GOES16/Products/CloudAndMoistureImagery/CONUS/Channel08/20211213/OR_ABI-L2-CMIPC-M6C08_G16_s20213472046170_e20213472046170_c20213472046170.nc
finished.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:49,891 [process_items]:
OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc:
12 of 15 tiles received.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:51,447 [process_items]: Finishing due to
timeout.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:51,556 [store_done]:
GOES16/Products/CloudAndMoistureImagery/CONUS/Channel13/20211213/OR_ABI-L2-CMIPC-M6C13_G16_s20213472046170_e20213472046170_c20213472046170.nc
finished.
<br>
<br>2) Hi Jim,
<br>
<br>I was just about to write my own email about this. It seems to me that not
all GOES-16 tiles are making it to NOAAPort. I was about to ask if anyone else
is seeing this, and I guess you are.
<br>
<br>I noticed this began around 14Z today. GOES-16 CONUS & Full-Disk are
impacted, Mesoscale and GOES-17 seem okay. I haven't seen any announcements
about data flow issues. Using notifyme I saw that Unidata was missing the same
tiles I was.
<br>
<br>Does anyone else have any insight?
<br>
<br>-Mike
<br>
<br>3) Hello NCO,
<br>
<br>Some GOES-16 tiles on NOAAport/SBN are missing. This started just after 14Z,
<br>when you awitched to the BMGS. See the thread below from a UNIDATA user log.
<br>Multiple people are seeing this.
<br>
<br>Gilbert Sebenste
<br>Meteorological Consultant
<br>AllisonHouse, LLC
<br>
<br>(SEC)</span></div></div></td></tr><tr><td><div style="padding:10px 15px;
border:1px solid lightgray; border-left:7px solid Purple;margin:2px
0;"><strong>2021-12-13 19:48:34 EST - Shellene Cope</strong><sup
style="float:right;">Additional comments</sup><div><span
style="word-wrap:break-word;display:block;">Jeff Kelley has emailed the
following reply: There has been no improvement in this issue. Have you heard
anything from NESDIS? It's strange that the imagery on the STAR and NASA
websites appear fine.
<br>
<br>https://weather.msfc.nasa.gov/cgi-bin/basicLooper.pl?category=goeseastabiconus&regex=10p35um&title=GOES-East%20ABI%20CONUS%20-%20%2010.35%20um%20-%20Band%2013&time_drop=show
<br>https://www.star.nesdis.noaa.gov/GOES/conus_band.php?sat=G16&band=13&length=24
<br>
<br>AWIPS and the FAA weather system continue to exhibit the same issue as this
link: https://weather.cod.edu/satrad/
<br>
<br>(SEC) </span></div></div></td></tr></tbody></table></div></div>
<div><p>The following work notes were given about the Incident:</p><div
style="margin-left:30px;"><table style="width:100%;"><tbody><tr><td><div
style="padding:10px 15px; border:1px solid lightgray; border-left:7px solid
gold;margin:2px 0;"><strong>2022-03-26 19:34:27 EDT - Chun Lam</strong><sup
style="float:right;">Work notes</sup><div><span
style="word-wrap:break-word;display:block;">[NCO-MCL] NCO is requesting via
email an update from Jeff Kelley to see if the issue still
exists.</span></div></div></td></tr><tr><td><div style="padding:10px 15px;
border:1px solid lightgray; border-left:7px solid gold;margin:2px
0;"><strong>2021-12-16 11:20:51 EST - Zheng Tzeng</strong><sup
style="float:right;">Work notes</sup><div><span
style="word-wrap:break-word;display:block;">reply from: jeff.kelley@xxxxxxxx
<br>
<br>Seeing the same thing at the CWSU in Indianapolis, in AWIPS and the FAA
<br>weather system.
<br>
<br>Jeff</span></div></div></td></tr><tr><td><div style="padding:10px 15px;
border:1px solid lightgray; border-left:7px solid gold;margin:2px
0;"><strong>2021-12-16 10:37:58 EST - Zheng Tzeng</strong><sup
style="float:right;">Work notes</sup><div><span
style="word-wrap:break-word;display:block;">reply from: gilbert@xxxxxxxxxxxxxxxx
<br>
<br>This issue has started again as of 14:30Z this
morning.</span></div></div></td></tr><tr><td><div style="padding:10px 15px;
border:1px solid lightgray; border-left:7px solid gold;margin:2px
0;"><strong>2021-12-13 20:39:36 EST - Curtis Sims</strong><sup
style="float:right;">Work notes</sup><div><span
style="word-wrap:break-word;display:block;">@0114z - OMB received an update
from gilbert@xxxxxxxxxxxxxxxx
<br>reply from: gilbert@xxxxxxxxxxxxxxxx
<br>
<br>Someone from the College of DuPage reached out to me with an interesting
observation from the University of Oklahoma...
<br>
<br>We had an outage on our NOAAport receive system this morning, which caused
our systems to fail over to our partner organization's feeds. On investigation,
we found that our receiver, ingest software, and LDM software were all working
fine. We saw packets coming into the Linux-based ingest server, but nothing was
being ingested. Then, it was noticed that the source IP of the NOAAport
multicast traffic had changed, for at least some of the packets. Whereas in the
past, it had been 10.0.9.51 , we are now seeing packets from 10.0.9.52 . Our
ingest server was not accepting those because it had no special routing rule
for that IP. We added such a rule (and did *not* delete the existing one for
the .51 IP) and our NOAAport ingest began working as before.
<br>
<br>I can confirm the source IP for me is 10.0.9.52 as well, but unlike him
that IP range was already open on my system and that traffic was making its way
in. I'm not sure if this is related or not, but I wanted to pass along the
intel in case it is.
<br>(CS)</span></div></div></td></tr><tr><td><div style="padding:10px 15px;
border:1px solid lightgray; border-left:7px solid gold;margin:2px
0;"><strong>2021-12-13 20:10:02 EST - Curtis Sims</strong><sup
style="float:right;">Work notes</sup><div><span
style="word-wrap:break-word;display:block;">@0054z - OMB received an update
from Jeff Kelley
<br>reply from: jeff.kelley@xxxxxxxx
<br>
<br>I just got off the phone with NCF. They are aware that data dropouts are
<br>occurring (with all data types, not just satellite) after switching between
<br>systems earlier today. The tech mentioned that it might be a firewall
<br>issue.
<br>
<br>Jeff
<br>(CS)</span></div></div></td></tr></tbody></table></div></div>
<hr>
<div> </div><div style="display:inline">Ref:MSG6116092</div></body></html>
