Re: [conduit] NCEP CONDUIT Services Fully Restored

• To: Dustin Sheffler - NOAA Federal <dustin.sheffler@xxxxxxxx>
• Subject: Re: [conduit] NCEP CONDUIT Services Fully Restored
• From: Pete Pokrandt <poker@xxxxxxxxxxxx>
• Date: Fri, 23 Sep 2022 20:31:27 +0000
• Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=aos.wisc.edu; dmarc=pass action=none header.from=aos.wisc.edu; dkim=pass header.d=aos.wisc.edu; arc=none
• Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=pLWDy4vyCGWzMEiWtZ0LY5Rw+BWfMgdMX5MxFUmlEyc=; b=SGThXIb6nQqQZTk1bSVq2FFoxNax9ay1CDtArXOoxdaU68801eTceYVILiZmodTLsu7iVzmO81gCB0reEWLTCDmgwrKvm2YgKYqhrJifTP3SQ2KtrEh6XSufr3Sq/oPMiNot30AqKj5/D0DGrul2CX9b380a9Xngt9FAdGvE1bdhrmJFFti4F53dZZk7GppPwZMGmbz4HXqrZHp89gBNRTGP+tFbHi52X58DbukgGyqVNiwDbyR69kMs7N4Kh5dZzBo79yWKiS3TuzLT1nTEMNT0LulYoO/eLGrqS4fqp2cwkJA4usf/q86fUNiOh5jsDlz30ZY6VuNofRbySJSJhw==
• Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mFrILDCEqO7a/FpCt9Bo7Zo23CUIjuiBHpvFjKO6ChoRTw0W3wHOysBFd+JWIgQkElUFHtIPfeV6zNHY0e7OEnAkQPoUGsbuE9bOyjT3Of6147FUPJMMALsexdgQ9XMo+zg8yn/xs2etu+QHpp/bd74JiN0PnfimC1b4yKbF0+MP5QmZyiW9v+I82szbdxNDWUhHX+OlojPM+EfJ7Y71H88+RZjarKafcQ7KeorW16ZnkRiWis5VnemXCRpfEH3jRMRJC1quAUuRboB/aOexetc+0nGln9Y8+GMitsB7g+PKyPhH+CNJOIf6/ZxtJ0AOYxV6/dO6/dKDqtdM37hEgQ==
• Authentication-results: dkim=none (message not signed) header.d=none; dmarc=none action=none header.from=aos.wisc.edu;
• Msip_labels:

Thanks, Dustin! Yes, the time looks correct after the reboot.

Have a good weekend!
Pete

Pete Pokrandt - System Engineer IV
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086  - poker@xxxxxxxxxxxx
________________________________
From: Dustin Sheffler - NOAA Federal <dustin.sheffler@xxxxxxxx>
Sent: Friday, September 23, 2022 3:11 PM
To: Pete Pokrandt <poker@xxxxxxxxxxxx>
Cc: support-conduit@xxxxxxxxxxxxxxxx <conduit@xxxxxxxxxxxxxxxx>
Subject: Re: [conduit] NCEP CONDUIT Services Fully Restored

Hi Pete,

Thanks for bringing this to our attention! I checked vm-lnx-conduit2 and 
confirmed that it was about 6 minutes ahead for its' system time. It seems the 
NTP that was enabled on this system to keep its' time synced is not functioning 
properly so I've opened a ticket to have our sys admins look at that. For now 
we've had the system rebooted which brought its' system time back in line with 
what it should be. Thanks again.

On Thu, Sep 22, 2022 at 4:33 PM Pete Pokrandt 
<poker@xxxxxxxxxxxx<mailto:poker@xxxxxxxxxxxx>> wrote:
Thanks, Dustin.

Just a quick note to let you know that it looks like the time on one of the 
main CONDUIT ingest machines 
(vm-lnx-conduit2.ncep.noaa.gov<http://vm-lnx-conduit2.ncep.noaa.gov>)  is 
drifting. We aren't losing any data right now, but that should probably be 
fixed sometime.

Pete


[cid:1836b9b98bbcb971f161]
https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+thredds.aos.wisc.edu


Pete Pokrandt - System Engineer IV
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086  - poker@xxxxxxxxxxxx<mailto:poker@xxxxxxxxxxxx>
________________________________
From: conduit 
<conduit-bounces@xxxxxxxxxxxxxxxx<mailto:conduit-bounces@xxxxxxxxxxxxxxxx>> on 
behalf of Dustin Sheffler - NOAA Federal via conduit 
<conduit@xxxxxxxxxxxxxxxx<mailto:conduit@xxxxxxxxxxxxxxxx>>
Sent: Wednesday, September 21, 2022 5:07 PM
To: support-conduit@xxxxxxxxxxxxxxxx<mailto:support-conduit@xxxxxxxxxxxxxxxx> 
<conduit@xxxxxxxxxxxxxxxx<mailto:conduit@xxxxxxxxxxxxxxxx>>
Subject: [conduit] NCEP CONDUIT Services Fully Restored

Conduit Users,

An overnight network issue had caused problems for many of our dataflows 
lasting into this afternoon as data caught back up. We had thought that CONDUIT 
services were restored by late morning but didn't notice until this afternoon 
that the grib insert process on our systems was continuing to fail. We finally 
restored data to NCEP Conduit servers around ~5:30pm EDT (2130 UTC). 
Unfortunately earlier model runs are not recoverable. We apologize for the 
inconvenience.

--
Dustin Sheffler
NCEP Central Operations - Dataflow
Email: dustin.sheffler@xxxxxxxx<mailto:dustin.sheffler@xxxxxxxx>
Google Voice: (301) 683-3827<tel:%28301%29%20683-1400>
<tel:%28301%29%20683-1400>


--
Dustin Sheffler
NCEP Central Operations - Dataflow
Email: dustin.sheffler@xxxxxxxx<mailto:dustin.sheffler@xxxxxxxx>
Google Voice: (301) 683-3827<tel:%28301%29%20683-1400>
<tel:%28301%29%20683-1400>

• Follow-Ups:
  • Re: [conduit] NCEP CONDUIT Services Fully Restored
    • From: Gilbert Sebenste

• References:
  • [conduit] NCEP CONDUIT Services Fully Restored
    • From: Dustin Sheffler - NOAA Federal
  • Re: [conduit] NCEP CONDUIT Services Fully Restored
    • From: Pete Pokrandt
  • Re: [conduit] NCEP CONDUIT Services Fully Restored
    • From: Dustin Sheffler - NOAA Federal