Re: [conduit] Conduit Outage 4/15 into 4/16

  • To: Mike Zuranski <mzuranski@xxxxxxxx>, CONDUIT Users List <conduit@xxxxxxxxxxxxxxxx>, LDM <ldm-users@xxxxxxxxxxxxxxxx>
  • Subject: Re: [conduit] Conduit Outage 4/15 into 4/16
  • From: Pete Pokrandt <poker@xxxxxxxxxxxx>
  • Date: Wed, 17 Apr 2024 16:55:40 +0000
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=aos.wisc.edu; dmarc=pass action=none header.from=aos.wisc.edu; dkim=pass header.d=aos.wisc.edu; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=tQG0wy91liOByBvLKJPcB6dSRFyYqjEAky0uOwh7nkQ=; b=WXLz4+Ydih2T3xcwcnesbIQfBEkgWgwvNsL1Jjd3okYPWYuMt+oIOrDh/QgBNRfgzL26BLFtcIjkYjtb+GvJg2/w9oMCDgBjyL+x7PT2vTXOVx5dy9A7pQti9Xk5Ipw3D5LGT/dS832dEhyaZeWNktmjnRT5WBu6V4hhYSXYKEJtL6yXMOCWP+qDRP7mGl2kJCsjUbVUmLIYzzGpYFXwE+0/jEsMlCtZxjGMuhxEy3h+yC44P9lrMI7mCz7z1n2Lm7soHtkLZG6TL8vPLkchX9erOMaBvT3Y9aBgO7b3dXu8OXGSagFIfdlVXKXqjndxAFFkaQK49AqE+t2UBrv1jA==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=m9H6F+GFw6rGCzV+NR3TFN/4J74s9YLAPzDpGh0xkT319sVhoIsQFINRYXAtrRHSTuvimbKuvB4PjWXohOuKPp2zhczlQL6udcukBwWk52wcc+TKudfBZmqJGt3i1G4dpGu3ecWP0uBAL0dNbum/uvzimKiYWIAG0JGR2Y5nLQLmqz/k2H5Qrnhqm7av4MUTVPEwSxSYV2eajzHS6vcNZAYD8CAZSGBce1/lgYtrNo5B5FYzegXv07lGkNCueOa9ItEWCtkrnn0vWj1UEIU6WZa8JNYhkn/BJHKPT/rtqXE4hd9G3sL5J/I53JXGyO8WKoAE1pCNsgTjaGJbY6KScA==
  • Msip_labels:
12 UTC GFS data seems to be coming in on CONDUIT. Fingers crossed..

Pete


Pete Pokrandt - System Engineer IV
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086  - poker@xxxxxxxxxxxx
________________________________
From: conduit <conduit-bounces@xxxxxxxxxxxxxxxx> on behalf of Mike Zuranski 
<mzuranski@xxxxxxxx>
Sent: Wednesday, April 17, 2024 9:47 AM
To: CONDUIT Users List <conduit@xxxxxxxxxxxxxxxx>; LDM 
<ldm-users@xxxxxxxxxxxxxxxx>
Subject: Re: [conduit] Conduit Outage 4/15 into 4/16

Good morning all,

The situation doesn't seem to have changed much from last night.  CONDUIT is 
still down, numerous impacts to NCEP web sites and services, cats & dogs living 
together...

We continue to monitor but that's about all we can do.  The link I'm smashing 
the F5 button on is this rtstats chart, it's probably the easiest "Is it still 
down?" check at the time for CONDUIT:
https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_num_nc?CONDUIT+conduit.unidata.ucar.edu<https://urldefense.com/v3/__https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_num_nc?CONDUIT*conduit.unidata.ucar.edu__;Kw!!Mak6IKo!I0UQhvkyqWnzDMjB9DrgkCzYa0NiAIqUUC31lwtA9MIk46SSSKORttcbuIzqBNff_NHcJjTmIs40nQ8jh4Bv$>

If you are having a hard time finding model data that is normally on CONDUIT, 
nomads.ncep.noaa.gov<https://urldefense.com/v3/__http://nomads.ncep.noaa.gov__;!!Mak6IKo!I0UQhvkyqWnzDMjB9DrgkCzYa0NiAIqUUC31lwtA9MIk46SSSKORttcbuIzqBNff_NHcJjTmIs40ndj-DX2A$>
 has remained unaffected by all of this.  Also, please reach out if you need 
assistance finding other sources of data or anything else.

Below is a notice that sums things up nicely; this and other notices can be 
found at 
https://www.nco.ncep.noaa.gov/status/messages/<https://urldefense.com/v3/__https://www.nco.ncep.noaa.gov/status/messages/__;!!Mak6IKo!I0UQhvkyqWnzDMjB9DrgkCzYa0NiAIqUUC31lwtA9MIk46SSSKORttcbuIzqBNff_NHcJjTmIs40nXRzxzBk$>
 (luckily that site has come back).  We will update with any changes to the 
situation.

SENIOR DUTY METEOROLOGIST NWS ADMINISTRATIVE MESSAGE
NWS NCEP CENTRAL OPERATIONS COLLEGE PARK MD
1019Z WED APR 17 2024


...UPDATES TO RECENT NWS OPERATIONAL OUTAGES...


WIDESPREAD WFO NETWORK OUTAGES...
WFOs internet and AWIPS connections have remained stable since
the circuit was restored in College Park Tuesday morning. NCO's
network team will continue to work towards mitigating impacts
during the recurring circuit outages in College Park.


NWS BROADCAST REPEATED MESSAGES...
The problem with NWS products being broadcasted multiple times
was traced to Monday's efforts to mitigate impacts from the
temperature spike in the College Park Data Center. NCO
implemented a fix to correct the issue at 2:00pm EDT Tuesday.


MRMS...
CONUS QPE data continues to not update on MRMS
(https://mrms.ncep.noaa.gov/data/<https://urldefense.com/v3/__https://mrms.ncep.noaa.gov/data/__;!!Mak6IKo!I0UQhvkyqWnzDMjB9DrgkCzYa0NiAIqUUC31lwtA9MIk46SSSKORttcbuIzqBNff_NHcJjTmIs40neAGapFo$>).
 The problem has been linked
to the College Park Data Center Outage. Efforts to restore the
data will resume early Wednesday.


RECOVERY EFFORTS IN THE COLLEGE PARK DATA CENTER...
No significant updates during overnight restoration efforts. NCEP
Center's (OPC, CPC, and WPC) operations remain severely degraded
due to downed NetApp systems in College Park. No ETR.

Current Known Impacts include:
-NCEP Centers' websites hosted in CP that remain inaccessible
include EMC and NCWCP intranet sites.
-WPC, OPC, and CPC's operational product suites' status, range
from being degraded to down.
-FTPPRD is inaccessible in CP (Customers are able to use
nomads.ncep.noaa.gov<https://urldefense.com/v3/__http://nomads.ncep.noaa.gov__;!!Mak6IKo!I0UQhvkyqWnzDMjB9DrgkCzYa0NiAIqUUC31lwtA9MIk46SSSKORttcbuIzqBNff_NHcJjTmIs40ndj-DX2A$>
 as a viable backup in the meantime)
-NCO operations personnel are unable monitor NWS networks and
circuits.
-CONUS QPE data is not updating on MRMS
(https://mrms.ncep.noaa.gov/data/<https://urldefense.com/v3/__https://mrms.ncep.noaa.gov/data/__;!!Mak6IKo!I0UQhvkyqWnzDMjB9DrgkCzYa0NiAIqUUC31lwtA9MIk46SSSKORttcbuIzqBNff_NHcJjTmIs40neAGapFo$>)
-Several layers are not updating on NWS Cloud Services (GIS and
Map Viewer)
-Multiple outside datasets are not available/delayed (UKMET data,
ECMWF data, Canadian METARS, ACARS aircraft data)




Gerhardt/SDM/NCO/NCEP


Best,
-Mike


Mike Zuranski
Data Engineer II
NSF Unidata Program Center
University Corporation for Atmospheric Research


On Tue, Apr 16, 2024 at 10:05 PM Mike Zuranski 
<mzuranski@xxxxxxxx<mailto:mzuranski@xxxxxxxx>> wrote:
The time is now 03:05 UTC, do you know where your data is?

At this time CONDUIT is still down.  I had an exchange with NCEP Ops earlier 
this evening so I know they were still working on it then.  They are fully 
aware of the breadth of the situation and are working with the applicable 
parties to resolve this as soon as possible.  Given that I do not plan on 
reaching out to them again on this unless the symptoms change.

I haven't seen any admin notices or similar pertaining to this, but we are 
watching closely.  I'll keep you posted with any updates I hear, and I plan on 
making another status update tomorrow morning.

In the meantime, the data that's missing on CONDUIT may well be found at 
nomads.ncep.noaa.gov<https://urldefense.com/v3/__http://nomads.ncep.noaa.gov__;!!Mak6IKo!I0UQhvkyqWnzDMjB9DrgkCzYa0NiAIqUUC31lwtA9MIk46SSSKORttcbuIzqBNff_NHcJjTmIs40ndj-DX2A$>,
 which has remained unaffected during these troubling times.  Feel free to 
reach out if we can help you find another source of data or anything else.

Best,
-Mike


Mike Zuranski
Data Engineer II
NSF Unidata Program Center
University Corporation for Atmospheric Research


On Tue, Apr 16, 2024 at 10:52 AM Mike Zuranski 
<mzuranski@xxxxxxxx<mailto:mzuranski@xxxxxxxx>> wrote:
Greetings all,

It appears the CONDUIT feed has been down between yesterday afternoon and just 
moments ago began to transmit data again.  It looks like this is being actively 
worked on but I'm keeping a close eye on the situation.

While I haven't seen any notifications on the subject, the source of the outage 
is upstream from us so we don't have much control over it.  If data stops 
flowing again I'll reach out to the appropriate parties.

Sorry for any inconvenience,
-Mike


Mike Zuranski
Data Engineer II
NSF Unidata Program Center
University Corporation for Atmospheric Research
  • 2024 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the conduit archives: