Yep. They switched back to the AMGS at 14:30Z this morning.
Gilbert
> On Dec 16, 2021, at 9:14 AM, Mike Zuranski <zuranski.wx@xxxxxxxxx> wrote:
>
>
> Hi all,
>
> This issue may have reappeared. Since 14Z this morning we're missing G16
> tiles on NOAAPort:
> https://weather.cod.edu/satrad/?parms=continental-conus-09-24-0-100-1&checked=map&colorbar=undefined
>
> I see Pete's data exhibiting the same symptoms:
> https://whirlwind.aos.wisc.edu/~wxp/goes16/wvc/goes16_conus.html
>
> I haven't seen any notifications on this yet, and the source IP in NP is
> still 10.0.9.51 FWIW.
>
> Happy Thursday,
> -Mike
>
> ======================
> Mike Zuranski
> Meteorology Support Analyst
> College of DuPage - Nexlab
> Weather.cod.edu
> ======================
>
>
>> On Tue, Dec 14, 2021 at 8:47 AM Weber, Jim <Jim.Weber@xxxxxxx> wrote:
>> All GOES data is looking complete again that I am receiving over the SBN.
>>
>>
>>
>> Jim
>>
>>
>>
>>
>>
>> From: Mike Zuranski <zuranski.wx@xxxxxxxxx>
>> Sent: Monday, December 13, 2021 10:33 PM
>> To: Gilbert Sebenste <gilbert@xxxxxxxxxxxxxxxx>
>> Cc: Weber, Jim <Jim.Weber@xxxxxxx>; LDM <ldm-users@xxxxxxxxxxxxxxxx>;
>> python-users@xxxxxxxxxxxxxxxx
>> Subject: Re: [ldm-users] [python-users] Goes Restitch Time Out
>>
>>
>>
>> Updates:
>>
>>
>>
>> SENIOR DUTY METEOROLOGIST NWS ADMINISTRATIVE MESSAGE
>> NWS NCEP CENTRAL OPERATIONS COLLEGE PARK MD
>> 0311Z TUE DEC 14 2021
>>
>>
>> ...MISSING/SHORT PRODUCTS SENT OUT OVER THE SBN...
>>
>> NCF has been investigating missing/short products being sent out
>> over the SBN today. They have just switched over the SBN uplink
>> to the HMGS which should resolve this issue.
>>
>>
>>
>> Since then the issue appears to have been resolved. Data flow looks to have
>> stabilized and our imagery looks normal.
>>
>>
>>
>>
>>
>> After the switch I took another look and the source IP on NOAAPort is back
>> to 10.0.9.51. Still no idea if it's related, but there's a data point.
>>
>>
>>
>> -Mike
>>
>>
>>
>> ======================
>>
>> Mike Zuranski
>>
>> Meteorology Support Analyst
>>
>> College of DuPage - Nexlab
>>
>> Weather.cod.edu
>>
>> ======================
>>
>>
>>
>>
>>
>> On Mon, Dec 13, 2021 at 5:50 PM Mike Zuranski <zuranski.wx@xxxxxxxxx> wrote:
>>
>> Thanks Gilbert.
>>
>>
>>
>> Someone from OU just reached out to me with an interesting observation...
>>
>>
>>
>> We had an outage on our NOAAport receive system this morning, which caused
>> our systems to fail over to our partner organization's feeds. On
>> investigation, we found that our receiver, ingest software, and LDM software
>> were all working fine. We saw packets coming into the Linux-based ingest
>> server, but nothing was being ingested. Then, it was noticed that the source
>> IP of the NOAAport multicast traffic had changed, for at least some of the
>> packets. Whereas in the past, it had been 10.0.9.51 , we are now seeing
>> packets from 10.0.9.52 . Our ingest server was not accepting those because
>> it had no special routing rule for that IP. We added such a rule (and did
>> *not* delete the existing one for the .51 IP) and our NOAAport ingest began
>> working as before.
>>
>>
>>
>> I can confirm the source IP for me is 10.0.9.52 as well, but unlike him that
>> IP range was already open on my system and that traffic was making its way
>> in. I'm not sure if this is related or not, but I wanted to pass along the
>> intel in case it is.
>>
>>
>>
>> -Mike
>>
>>
>>
>> ======================
>>
>> Mike Zuranski
>>
>> Meteorology Support Analyst
>>
>> College of DuPage - Nexlab
>>
>> Weather.cod.edu
>>
>> ======================
>>
>>
>>
>>
>>
>> On Mon, Dec 13, 2021 at 5:07 PM Gilbert Sebenste <gilbert@xxxxxxxxxxxxxxxx>
>> wrote:
>>
>> I have put in a trouble ticket with NCO. I'll keep everyone posted.
>>
>>
>>
>> Gilbert
>>
>>
>>
>>
>> On Dec 13, 2021, at 3:11 PM, Mike Zuranski <zuranski.wx@xxxxxxxxx> wrote:
>>
>>
>>
>> Hi Jim,
>>
>>
>>
>> I was just about to write my own email about this. It seems to me that not
>> all GOES-16 tiles are making it to NOAAPort. I was about to ask if anyone
>> else is seeing this, and I guess you are.
>>
>>
>>
>> I noticed this began around 14Z today. GOES-16 CONUS & Full-Disk are
>> impacted, Mesoscale and GOES-17 seem okay. I haven't seen any announcements
>> about data flow issues. Using notifyme I saw that Unidata was missing the
>> same tiles I was.
>>
>>
>>
>> Does anyone else have any insight?
>>
>>
>>
>> -Mike
>>
>>
>>
>> ======================
>>
>> Mike Zuranski
>>
>> Meteorology Support Analyst
>>
>> College of DuPage - Nexlab
>>
>> Weather.cod.edu
>>
>> ======================
>>
>>
>>
>>
>>
>> On Mon, Dec 13, 2021 at 3:06 PM Weber, Jim <Jim.Weber@xxxxxxx> wrote:
>>
>> I have been using the GOES Restitch for a couple of months now and it has
>> worked perfectly until today. Today I am getting partial images. Upon
>> looking at the log files I am finding that it is “Timing Out” waiting for
>> the tile to arrive. I am not seeing any large amounts of dropped packets. I
>> am not sure what could be causing my sudden issue. Any insight as to what I
>> might check?
>>
>>
>>
>> Thanks, Jim ( partial log below)
>>
>>
>>
>>
>>
>> [TIRE 132051] 2021-12-13 20:54:52,378 [process_items]:
>> OR_ABI-L2-CMIPC-M6C05_G16_s20213472051170_e20213472051170_c20213472051170.nc:
>> 8 of 15 tiles received.
>>
>> [TIRE 132051] 2021-12-13 20:54:53,177 [process_items]:
>> OR_ABI-L2-CMIPC-M6C01_G16_s20213472051170_e20213472051170_c20213472051170.nc:
>> 7 of 15 tiles received.
>>
>> [TIRE 132051] 2021-12-13 20:54:54,033 [process_items]:
>> OR_ABI-L2-CMIPC-M6C01_G16_s20213472051170_e20213472051170_c20213472051170.nc:
>> 8 of 15 tiles received.
>>
>> [TIRE 132051] 2021-12-13 20:55:00,091 [process_items]:
>> OR_ABI-L2-CMIPC-M6C03_G16_s20213472051170_e20213472051170_c20213472051170.nc:
>> 11 of 15 tiles received.
>>
>> [TIRE 132051] 2021-12-13 20:55:00,753 [process_items]:
>> OR_ABI-L2-CMIPC-M6C01_G16_s20213472051170_e20213472051170_c20213472051170.nc:
>> 9 of 15 tiles received.
>>
>> [TIRE 132051] 2021-12-13 20:55:01,554 [process_items]:
>> OR_ABI-L2-CMIPC-M6C03_G16_s20213472051170_e20213472051170_c20213472051170.nc:
>> 12 of 15 tiles received.
>>
>> [TIRE 132051] 2021-12-13 20:55:06,066 [process_items]:
>> OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc:
>> 6 of 15 tiles received.
>>
>> [TIRE 132051] 2021-12-13 20:55:06,350 [process_items]:
>> OR_ABI-L2-CMIPC-M6C03_G16_s20213472051170_e20213472051170_c20213472051170.nc:
>> 13 of 15 tiles received.
>>
>> [TIRE 132051] 2021-12-13 20:55:07,089 [process_items]:
>> OR_ABI-L2-CMIPC-M6C05_G16_s20213472051170_e20213472051170_c20213472051170.nc:
>> 9 of 15 tiles received.
>>
>> [TIRE 132051] 2021-12-13 20:55:07,592 [process_items]:
>> OR_ABI-L2-CMIPC-M6C01_G16_s20213472051170_e20213472051170_c20213472051170.nc:
>> 10 of 15 tiles received.
>>
>> [TIRE 132051] 2021-12-13 20:55:08,514 [process_items]:
>> OR_ABI-L2-CMIPC-M6C05_G16_s20213472051170_e20213472051170_c20213472051170.nc:
>> 10 of 15 tiles received.
>>
>> [TIRE 132051] 2021-12-13 20:55:13,149 [process_items]:
>> OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc:
>> 7 of 15 tiles received.
>>
>> [TIRE 132051] 2021-12-13 20:55:16,471 [process_items]:
>> OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc:
>> 8 of 15 tiles received.
>>
>> [TIRE 132051] 2021-12-13 20:55:22,188 [process_items]:
>> OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc:
>> 9 of 15 tiles received.
>>
>> [TIRE 132051] 2021-12-13 20:55:34,664 [process_items]:
>> OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc:
>> 10 of 15 tiles received.
>>
>> [TIRE 132046] 2021-12-13 20:55:44,834 [process_items]: Finishing due to
>> timeout.
>>
>> [TIRE 132046] 2021-12-13 20:55:44,944 [store_done]:
>> GOES16/Products/CloudAndMoistureImagery/CONUS/Channel07/20211213/OR_ABI-L2-CMIPC-M6C07_G16_s20213472046170_e20213472046170_c20213472046170.nc
>> finished.
>>
>> [TIRE 132046] 2021-12-13 20:55:44,947 [process_items]: Finishing due to
>> timeout.
>>
>> [TIRE 132046] 2021-12-13 20:55:45,051 [store_done]:
>> GOES16/Products/CloudAndMoistureImagery/CONUS/Channel12/20211213/OR_ABI-L2-CMIPC-M6C12_G16_s20213472046170_e20213472046170_c20213472046170.nc
>> finished.
>>
>> [TIRE 132051] 2021-12-13 20:55:45,698 [process_items]:
>> OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc:
>> 11 of 15 tiles received.
>>
>> [TIRE 132046] 2021-12-13 20:55:47,772 [process_items]: Finishing due to
>> timeout.
>>
>> [TIRE 132046] 2021-12-13 20:55:47,875 [store_done]:
>> GOES16/Products/CloudAndMoistureImagery/CONUS/Channel10/20211213/OR_ABI-L2-CMIPC-M6C10_G16_s20213472046170_e20213472046170_c20213472046170.nc
>> finished.
>>
>> [TIRE 132046] 2021-12-13 20:55:48,075 [process_items]: Finishing due to
>> timeout.
>>
>> [TIRE 132046] 2021-12-13 20:55:48,181 [store_done]:
>> GOES16/Products/CloudAndMoistureImagery/CONUS/Channel11/20211213/OR_ABI-L2-CMIPC-M6C11_G16_s20213472046170_e20213472046170_c20213472046170.nc
>> finished.
>>
>> [TIRE 132046] 2021-12-13 20:55:49,455 [process_items]: Finishing due to
>> timeout.
>>
>> [TIRE 132046] 2021-12-13 20:55:49,559 [store_done]:
>> GOES16/Products/CloudAndMoistureImagery/CONUS/Channel09/20211213/OR_ABI-L2-CMIPC-M6C09_G16_s20213472046170_e20213472046170_c20213472046170.nc
>> finished.
>>
>> [TIRE 132046] 2021-12-13 20:55:49,560 [process_items]: Finishing due to
>> timeout.
>>
>> [TIRE 132046] 2021-12-13 20:55:49,664 [store_done]:
>> GOES16/Products/CloudAndMoistureImagery/CONUS/Channel08/20211213/OR_ABI-L2-CMIPC-M6C08_G16_s20213472046170_e20213472046170_c20213472046170.nc
>> finished.
>>
>> [TIRE 132051] 2021-12-13 20:55:49,891 [process_items]:
>> OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc:
>> 12 of 15 tiles received.
>>
>> [TIRE 132046] 2021-12-13 20:55:51,447 [process_items]: Finishing due to
>> timeout.
>>
>> [TIRE 132046] 2021-12-13 20:55:51,556 [store_done]:
>> GOES16/Products/CloudAndMoistureImagery/CONUS/Channel13/20211213/OR_ABI-L2-CMIPC-M6C13_G16_s20213472046170_e20213472046170_c20213472046170.nc
>> finished.
>>
>>
>>
>>
>>
>> This message and its attachments may contain legally privileged or
>> confidential information. It is intended solely for the named addressee(s).
>> If you are not an addressee indicated in this message (or responsible for
>> delivery of the message to an addressee), you may not copy or deliver this
>> message or its attachments to anyone. Rather, you should permanently delete
>> this message and its attachments and kindly notify the sender by reply
>> e-mail. Any content of this message and its attachments that does not relate
>> to the official business of Fox Corporation, or its subsidiaries must be
>> taken not to have been sent or endorsed by any of them. No representation is
>> made that this email or its attachments are without defect.
>>
>> _______________________________________________
>> NOTE: All exchanges posted to Unidata maintained email lists are
>> recorded in the Unidata inquiry tracking system and made publicly
>> available through the web. Users who post to any of the lists we
>> maintain are reminded to remove any personal information that they
>> do not want to be made public.
>>
>>
>> python-users mailing list
>> python-users@xxxxxxxxxxxxxxxx
>> For list information, to unsubscribe, or change your membership options,
>> visit: https://www.unidata.ucar.edu/mailing_lists/
>>
>> _______________________________________________
>> NOTE: All exchanges posted to Unidata maintained email lists are
>> recorded in the Unidata inquiry tracking system and made publicly
>> available through the web. Users who post to any of the lists we
>> maintain are reminded to remove any personal information that they
>> do not want to be made public.
>>
>>
>> ldm-users mailing list
>> ldm-users@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> https://www.unidata.ucar.edu/mailing_lists/
>>
>>
>>
>> This message and its attachments may contain legally privileged or
>> confidential information. It is intended solely for the named addressee(s).
>> If you are not an addressee indicated in this message (or responsible for
>> delivery of the message to an addressee), you may not copy or deliver this
>> message or its attachments to anyone. Rather, you should permanently delete
>> this message and its attachments and kindly notify the sender by reply
>> e-mail. Any content of this message and its attachments that does not relate
>> to the official business of Fox Corporation, or its subsidiaries must be
>> taken not to have been sent or endorsed by any of them. No representation is
>> made that this email or its attachments are without defect.
