Hi Pete,
As of yesterday we failed almost all of our applications to our site in
Boulder (meaning away from CONDUIT). Have you noticed an improvement in
your speeds since yesterday afternoon? If so this will give us a clue that
maybe there's something interfering on our side that isn't specifically
CONDUIT, but another app that might be causing congestion. (And if it's the
same then that's a clue in the other direction.)
Thanks,
Anne
On Mon, Apr 1, 2019 at 3:24 PM Pete Pokrandt <poker@xxxxxxxxxxxx> wrote:
> The lag here at UW-Madison was up to 1200 seconds today, and that's with a
> 10-way split feed. Whatever is causing the issue has definitely not been
> resolved, and historically is worse during the work week than on the
> weekends. If that helps at all.
>
> Pete
>
>
>
> <http://www.weather.com/tv/shows/wx-geeks/video/the-incredible-shrinking-cold-pool>
> --
> Pete Pokrandt - Systems Programmer
> UW-Madison Dept of Atmospheric and Oceanic Sciences
> 608-262-3086 - poker@xxxxxxxxxxxx
>
> ------------------------------
> *From:* Anne Myckow - NOAA Affiliate <anne.myckow@xxxxxxxx>
> *Sent:* Thursday, March 28, 2019 4:28 PM
> *To:* Person, Arthur A.
> *Cc:* Carissa Klemmer - NOAA Federal; Pete Pokrandt;
> _NCEP.List.pmb-dataflow; conduit@xxxxxxxxxxxxxxxx;
> support-conduit@xxxxxxxxxxxxxxxx
> *Subject:* Re: [Ncep.list.pmb-dataflow] Large lags on CONDUIT feed -
> started a week or so ago
>
> Hello Art,
>
> We will not be upgrading to version 6.13 on these systems as they are not
> robust enough to support the local logging inherent in the new version.
>
> I will check in with my team on if there are any further actions we can
> take to try and troubleshoot this issue, but I fear we may be at the limit
> of our ability to make this better.
>
> I’ll let you know tomorrow where we stand. Thanks.
> Anne
>
> On Mon, Mar 25, 2019 at 3:00 PM Person, Arthur A. <aap1@xxxxxxx> wrote:
>
> Carissa,
>
>
> Can you report any status on this inquiry?
>
>
> Thanks... Art
>
>
> Arthur A. Person
> Assistant Research Professor, System Administrator
> Penn State Department of Meteorology and Atmospheric Science
> email: aap1@xxxxxxx, phone: 814-863-1563 <callto:814-863-1563>
>
>
> ------------------------------
> *From:* Carissa Klemmer - NOAA Federal <carissa.l.klemmer@xxxxxxxx>
> *Sent:* Tuesday, March 12, 2019 8:30 AM
> *To:* Pete Pokrandt
> *Cc:* Person, Arthur A.; conduit@xxxxxxxxxxxxxxxx;
> support-conduit@xxxxxxxxxxxxxxxx; _NCEP.List.pmb-dataflow
> *Subject:* Re: Large lags on CONDUIT feed - started a week or so ago
>
> Hi Everyone
>
> I’ve added the Dataflow team email to the thread. I haven’t heard that any
> changes were made or that any issues were found. But the team can look
> today and see if we have any signifiers of overall slowness with anything .
>
> Dataflow, try taking a look at the new Citrix or VM troubleshooting tools
> if there are any abnormal signatures that may explain this.
>
> On Monday, March 11, 2019, Pete Pokrandt <poker@xxxxxxxxxxxx> wrote:
>
> Art,
>
> I don't know if NCEP ever figured anything out, but I've been able to keep
> my latencies reasonable (300-600s max, mostly during the 12 UTC model
> suite) by splitting my CONDUIT request 10 ways, instead of the 5 that I had
> been doing, or in a single request. Maybe give that a try and see if it
> helps at all.
>
> Pete
>
>
>
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.weather.com%2Ftv%2Fshows%2Fwx-geeks%2Fvideo%2Fthe-incredible-shrinking-cold-pool&data=02%7C01%7Caap1%40psu.edu%7C5bc029df8d3b49fe895008d6a6e67860%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636879906133526700&sdata=rISFh7zNHrOY%2B4K4qBEqwtB2zq%2B0IE4l%2FmqYXRAcM6A%3D&reserved=0>
> --
> Pete Pokrandt - Systems Programmer
> UW-Madison Dept of Atmospheric and Oceanic Sciences
> 608-262-3086 - poker@xxxxxxxxxxxx
>
> ------------------------------
> *From:* Person, Arthur A. <aap1@xxxxxxx>
> *Sent:* Monday, March 11, 2019 3:45 PM
> *To:* Holly Uhlenhake - NOAA Federal; Pete Pokrandt
> *Cc:* conduit@xxxxxxxxxxxxxxxx; support-conduit@xxxxxxxxxxxxxxxx
> *Subject:* Re: [conduit] Large lags on CONDUIT feed - started a week or
> so ago
>
>
> Holly,
>
>
> Was there any resolution to this on the NCEP end? I'm still seeing
> terrible delays (1000-4000 seconds) receiving data from
> conduit.ncep.noaa.gov
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fconduit.ncep.noaa.gov&data=02%7C01%7Caap1%40psu.edu%7C5bc029df8d3b49fe895008d6a6e67860%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636879906133536705&sdata=5DIx0hfLu7Q1FxggAkcsn3845ayF0rfUUbnAADYjblY%3D&reserved=0>.
> It would be helpful to know if things are resolved at NCEP's end so I know
> whether to look further down the line.
>
>
> Thanks... Art
>
>
> Arthur A. Person
> Assistant Research Professor, System Administrator
> Penn State Department of Meteorology and Atmospheric Science
> email: aap1@xxxxxxx, phone: 814-863-1563 <callto:814-863-1563>
>
>
> ------------------------------
> *From:* conduit-bounces@xxxxxxxxxxxxxxxx <conduit-bounces@xxxxxxxxxxxxxxxx>
> on behalf of Holly Uhlenhake - NOAA Federal <holly.uhlenhake@xxxxxxxx>
> *Sent:* Thursday, February 21, 2019 12:05 PM
> *To:* Pete Pokrandt
> *Cc:* conduit@xxxxxxxxxxxxxxxx; support-conduit@xxxxxxxxxxxxxxxx
> *Subject:* Re: [conduit] Large lags on CONDUIT feed - started a week or
> so ago
>
> Hi Pete,
>
> We'll take a look and see if we can figure out what might be going on. We
> haven't done anything to try and address this yet, but based on your
> analysis I'm suspicious that it might be tied to a resource constraint on
> the VM or the blade it resides on.
>
> Thanks,
> Holly Uhlenhake
> Acting Dataflow Team Lead
>
> On Thu, Feb 21, 2019 at 11:32 AM Pete Pokrandt <poker@xxxxxxxxxxxx> wrote:
>
> Just FYI, data is flowing, but the large lags continue.
>
>
> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd.aos.wisc.edu
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Frtstats.unidata.ucar.edu%2Fcgi-bin%2Frtstats%2Fiddstats_nc%3FCONDUIT%2Bidd.aos.wisc.edu&data=02%7C01%7Caap1%40psu.edu%7C5bc029df8d3b49fe895008d6a6e67860%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636879906133536705&sdata=KqSHAeB6XYUOR2tVzRRSdMCzVahKVPOiPzvC9xoHMBU%3D&reserved=0>
>
> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+conduit.unidata.ucar.edu
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Frtstats.unidata.ucar.edu%2Fcgi-bin%2Frtstats%2Fiddstats_nc%3FCONDUIT%2Bconduit.unidata.ucar.edu&data=02%7C01%7Caap1%40psu.edu%7C5bc029df8d3b49fe895008d6a6e67860%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636879906133546714&sdata=I1mvCqaacew8DEknYPh3%2Fd69L6K2i6CtIKpgHDz9GT8%3D&reserved=0>
>
> Pete
>
>
>
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.weather.com%2Ftv%2Fshows%2Fwx-geeks%2Fvideo%2Fthe-incredible-shrinking-cold-pool&data=02%7C01%7Caap1%40psu.edu%7C5bc029df8d3b49fe895008d6a6e67860%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636879906133546714&sdata=pvl82se6O%2Fvtzi4DBKM%2F0YrJALvDoPE4k3Nz9hSUyyk%3D&reserved=0>
> --
> Pete Pokrandt - Systems Programmer
> UW-Madison Dept of Atmospheric and Oceanic Sciences
> 608-262-3086 - poker@xxxxxxxxxxxx
>
> ------------------------------
> *From:* conduit-bounces@xxxxxxxxxxxxxxxx <conduit-bounces@xxxxxxxxxxxxxxxx>
> on behalf of Pete Pokrandt <poker@xxxxxxxxxxxx>
> *Sent:* Wednesday, February 20, 2019 12:07 PM
> *To:* Carissa Klemmer - NOAA Federal
> *Cc:* conduit@xxxxxxxxxxxxxxxx; support-conduit@xxxxxxxxxxxxxxxx
> *Subject:* Re: [conduit] Large lags on CONDUIT feed - started a week or
> so ago
>
> Data is flowing again - picked up somewhere in the GEFS. Maybe CONDUIT
> server was restarted, or ldm on it? Lags are large (3000s+) but dropping
> slowly
>
> Pete
>
>
>
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.weather.com%2Ftv%2Fshows%2Fwx-geeks%2Fvideo%2Fthe-incredible-shrinking-cold-pool&data=02%7C01%7Caap1%40psu.edu%7C5bc029df8d3b49fe895008d6a6e67860%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636879906133556719&sdata=zcmqQMKWWa3YXEiNQyWYJFCPWCMFrR1kDC%2B8jMZgTuY%3D&reserved=0>
> --
> Pete Pokrandt - Systems Programmer
> UW-Madison Dept of Atmospheric and Oceanic Sciences
> 608-262-3086 - poker@xxxxxxxxxxxx
>
> ------------------------------
> *From:* conduit-bounces@xxxxxxxxxxxxxxxx <conduit-bounces@xxxxxxxxxxxxxxxx>
> on behalf of Pete Pokrandt <poker@xxxxxxxxxxxx>
> *Sent:* Wednesday, February 20, 2019 11:56 AM
> *To:* Carissa Klemmer - NOAA Federal
> *Cc:* conduit@xxxxxxxxxxxxxxxx; support-conduit@xxxxxxxxxxxxxxxx
> *Subject:* Re: [conduit] Large lags on CONDUIT feed - started a week or
> so ago
>
> Just a quick follow-up - we started falling far enough behind (3600+ sec)
> that we are losing data. We got short files starting at 174h into the GFS
> run, and only got (incomplete) data through 207h.
>
> We have now not received any data on CONDUIT since 11:27 AM CST (1727 UTC)
> today (Wed Feb 20)
>
> Pete
>
>
>
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.weather.com%2Ftv%2Fshows%2Fwx-geeks%2Fvideo%2Fthe-incredible-shrinking-cold-pool&data=02%7C01%7Caap1%40psu.edu%7C5bc029df8d3b49fe895008d6a6e67860%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636879906133556719&sdata=zcmqQMKWWa3YXEiNQyWYJFCPWCMFrR1kDC%2B8jMZgTuY%3D&reserved=0>
> --
> Pete Pokrandt - Systems Programmer
> UW-Madison Dept of Atmospheric and Oceanic Sciences
> 608-262-3086 - poker@xxxxxxxxxxxx
>
> ------------------------------
> *From:* conduit-bounces@xxxxxxxxxxxxxxxx <conduit-bounces@xxxxxxxxxxxxxxxx>
> on behalf of Pete Pokrandt <poker@xxxxxxxxxxxx>
> *Sent:* Wednesday, February 20, 2019 11:28 AM
> *To:* Carissa Klemmer - NOAA Federal
> *Cc:* conduit@xxxxxxxxxxxxxxxx; support-conduit@xxxxxxxxxxxxxxxx
> *Subject:* [conduit] Large lags on CONDUIT feed - started a week or so ago
>
> Carissa,
>
> We have been feeding CONDUIT using a 5 way split feed direct from
> conduit.ncep.noaa.gov
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fconduit.ncep.noaa.gov&data=02%7C01%7Caap1%40psu.edu%7C5bc029df8d3b49fe895008d6a6e67860%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636879906133566724&sdata=zeEqXgG81kDQIcbzVlj78FXdyz2ld1LKhmVaYP9bA1M%3D&reserved=0>,
> and it had been really good for some time, lags 30-60 seconds or less.
>
> However, the past week or so, we've been seeing some very large lags
> during each 6 hour model suite - Unidata is also seeing these - they are
> also feeding direct from conduit.ncep.noaa.gov
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fconduit.ncep.noaa.gov&data=02%7C01%7Caap1%40psu.edu%7C5bc029df8d3b49fe895008d6a6e67860%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636879906133576734&sdata=yPlWPGkRjNuMQUHK6ZMz2CSOnt5GnyP0N5WBJ2Xl0n4%3D&reserved=0>
> .
>
>
> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd.aos.wisc.edu
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Frtstats.unidata.ucar.edu%2Fcgi-bin%2Frtstats%2Fiddstats_nc%3FCONDUIT%2Bidd.aos.wisc.edu&data=02%7C01%7Caap1%40psu.edu%7C5bc029df8d3b49fe895008d6a6e67860%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636879906133576734&sdata=g%2FWWKnNIJNPT4MRavn0YLNAUlKVBFl49%2Beipn5ftYkw%3D&reserved=0>
>
>
> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+conduit.unidata.ucar.edu
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Frtstats.unidata.ucar.edu%2Fcgi-bin%2Frtstats%2Fiddstats_nc%3FCONDUIT%2Bconduit.unidata.ucar.edu&data=02%7C01%7Caap1%40psu.edu%7C5bc029df8d3b49fe895008d6a6e67860%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636879906133586734&sdata=p%2F1GcWu%2FAJUQwg1qWexK5DgSQCEjUn1T2udhtx4fix8%3D&reserved=0>
>
>
> Any idea what's going on, or how we can find out?
>
> Thanks!
> Pete
>
>
>
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.weather.com%2Ftv%2Fshows%2Fwx-geeks%2Fvideo%2Fthe-incredible-shrinking-cold-pool&data=02%7C01%7Caap1%40psu.edu%7C5bc029df8d3b49fe895008d6a6e67860%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636879906133596744&sdata=eFcuheAVvxEYMwwGes9fTTPts9sdGjFA2pQ0rpcDAdw%3D&reserved=0>
> --
> Pete Pokrandt - Systems Programmer
> UW-Madison Dept of Atmospheric and Oceanic Sciences
> 608-262-3086 - poker@xxxxxxxxxxxx
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> conduit mailing list
> conduit@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.unidata.ucar.edu%2Fmailing_lists%2F&data=02%7C01%7Caap1%40psu.edu%7C5bc029df8d3b49fe895008d6a6e67860%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C636879906133596744&sdata=kJgg9477GhOS4LXezS9HI6%2B%2BbkhqVgKhmtBmBXddoK4%3D&reserved=0>
>
>
>
> --
> Carissa Klemmer
> NCEP Central Operations
> IDSB Branch Chief
> 301-683-3835
>
> _______________________________________________
> Ncep.list.pmb-dataflow mailing list
> Ncep.list.pmb-dataflow@xxxxxxxxxxxxxxxxxxxx
> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.pmb-dataflow
>
> --
> Anne Myckow
> Lead Dataflow Analyst
> NOAA/NCEP/NCO
> 301-683-3825
>
--
Anne Myckow
Lead Dataflow Analyst
NOAA/NCEP/NCO
301-683-3825
