Re: [ldm-users] NEXRAD2/NEXRAD3 lag?

• To: Blair Trosper <blair.trosper@xxxxxxxxxxxxxxxxxxx>
• Subject: Re: [ldm-users] NEXRAD2/NEXRAD3 lag?
• From: Pete Pokrandt <poker@xxxxxxxxxxxx>
• Date: Sun, 18 May 2014 20:43:19 -0500

Blair,

[ldm@idd ~]$ ldmadmin watch -f nexrad2
(Type ^D when finished)

Pete

On 5/18/2014 8:24 PM, Blair Trosper wrote:

We're not the only ones seeing it...and again, all our servers are 1Gbps using less than 1% of their bandwidth.

It's also for ALL sites, not any specific WFO/SRH. I've checked with at least two of our current providers (who pull from the very top of the topology, and they confirm the issue).

Karen: Can you nominate an upstream server for one of ours to use so we can test this out? Can we be fed from a tier 1 server for a day or two to definitely rule that out?

I might be with you if it were one server, but we're talking about gigabit servers on separate backbone providers with separate upstream providers in four geographically disparate locations...all with the */exact/* same lag at any given moment. The odds of that are astronomical, especially when you factor in that one tier-1-minutes-one providers see the same thing.

On Sat, May 17, 2014 at 9:35 AM, Karen Cooper - NOAA Affiliate <karen.cooper@xxxxxxxx <mailto:karen.cooper@xxxxxxxx>> wrote:

    The level 2 data is inserted into Ldm queues at the radar RPGs as
    it is generated.   If it were a problem there everyone would be
    seeing it.

    Sent from me!  Please pardon any tpyos


    On May 17, 2014, at 1:18 AM, Blair Trosper
    <blair.trosper@xxxxxxxxxxxxxxxxxxx
    <mailto:blair.trosper@xxxxxxxxxxxxxxxxxxx>> wrote:

    No joy there.  Our providers (TAMU, OU, WISC, et al) are all
    under tier 1 providers, and there is no lag.

    I can't find the document that describes how the NEXRAD2 data is
    fed into the LDM system.  I think it happens in DC at the
    Internet2 aperture, but that information may be out of date (or I
    may be remembering it wrong).

    As I recall, two weeks ago, we weren't the only ones seeing this lag.

    And we're still seeing it on three servers with three separate
    upstream providers -- and each server is deliberately on
    different BGP/backbone providers for geographic AND network
    diversity.

    Something's not quite right, and I think the key is understanding
    how the NEXRAD2 data gets into LDM...and then going to THAT
    source to find out why it's delayed.  I know Allison House's data
    isn't delayed (whereas iastate's is), so my guess is they're
    acquiring it independently of LDM.

    So we know iastate.edu <http://iastate.edu> is getting data 10
    minutes late just like we are, as are our upstream
    providers...but the servers themselves are always at 0 or 1
    second of lag.

    Something is definitely wrong.  We don't see any lag on any of
    the other feed trees (HDS, IDS, WMO, NGRID, UNIWISC, NEXRAD3,
    etc).  This seems to be unique to NEXRAD2.

    Can someone from UCAR/UNIDATA jump in and help us...or should we
    open a formal support ticket?


    On Sun, May 11, 2014 at 5:49 PM, Steven Emmerson
    <emmerson.steven@xxxxxxxxx <mailto:emmerson.steven@xxxxxxxxx>> wrote:

        It could be that the last NEXRAD2 data-product is stuck in a
        system buffer. Try putting a "-flush" or "-close" on the
        relevant PIPE action.

        On May 11, 2014 11:49 AM, "Gilbert Sebenste"
        <sebenste@xxxxxxxxxxxxxxxxxxxxx
        <mailto:sebenste@xxxxxxxxxxxxxxxxxxxxx>> wrote:

            Hi Blair,

                We are still seeing this.  I know the TOC was made
                aware of this issue two
                weeks ago, but despite all our servers being 0
                seconds lagged on the
                NEXRAD2 feed tree, our data (from multiple providers
                who are also not
                lagged) is at least 10 minutes behind.


            Been off the grid, for the most part, or I would have
            answered earlier. Things to look for:

            Can you do a traceroute to the servers in question? See
            if anything is
            jammed up inbetween them and you.

            Are all network cards set at least at 100 mb? If 10 mb,
            it can't handle it.

            Is the system clock correct? Are you using ntp? This is a
            must.

                Also, this page is a 500 internal server error and
                has been for months, and nobody has fixed it or seems
                to even notice:
                
http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/rtstats_topogif?NEXRAD2

                And this one takes over a minute to load:
                
http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/rtstats_feedtree?NEXRAD2

                ...but you can verify from the second link that we're
                not lagging (search
                for "tampa" or "chicago").


            Give those a try aboove, and see what happens.

            Gilbert

            
*******************************************************************************
            Gilbert Sebenste  ********
            (My opinions only!)  ******
            Staff Meteorologist, Northern Illinois University      ****
            E-mail: sebenste@xxxxxxxxxxxxxxxxxxxxx

<mailto:sebenste@xxxxxxxxxxxxxxxxxxxxx> ***

            web: http://weather.admin.niu.edu  **

Twitter: http://www.twitter.com/NIU_Weather ** Facebook: http://www.facebook.com/niu.weather *

            
*******************************************************************************

            _______________________________________________
            ldm-users mailing list
            ldm-users@xxxxxxxxxxxxxxxx
            <mailto:ldm-users@xxxxxxxxxxxxxxxx>
            For list information or to unsubscribe,  visit:
            http://www.unidata.ucar.edu/mailing_lists/


        _______________________________________________
        ldm-users mailing list
        ldm-users@xxxxxxxxxxxxxxxx <mailto:ldm-users@xxxxxxxxxxxxxxxx>
        For list information or to unsubscribe,  visit:
        http://www.unidata.ucar.edu/mailing_lists/

--

    ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
    ?? ?? ?? ?? ?? ?? ?? ?? ?
    Blair Trosper
    Updraft Networks / Weather Data
      NOC: 469-844-5440 <tel:469-844-5440>
      Early Watch Notifications: http://twitter.com/weatherwatches
    _______________________________________________
    ldm-users mailing list
    ldm-users@xxxxxxxxxxxxxxxx <mailto:ldm-users@xxxxxxxxxxxxxxxx>
    For list information or to unsubscribe,  visit:
    http://www.unidata.ucar.edu/mailing_lists/

--

? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ?? ?? ?? ?? ?? ?? ?? ?

Blair Trosper
  Updraft Networks / Weather Data
  NOC:  469-844-5440
  Early Watch Notifications: http://twitter.com/weatherwatches


_______________________________________________
ldm-users mailing list
ldm-users@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/

• References:
  • [ldm-users] NEXRAD2/NEXRAD3 lag?
    • From: Blair Trosper
  • Re: [ldm-users] NEXRAD2/NEXRAD3 lag?
    • From: Blair Trosper
  • Re: [ldm-users] NEXRAD2/NEXRAD3 lag?
    • From: Gilbert Sebenste
  • Re: [ldm-users] NEXRAD2/NEXRAD3 lag?
    • From: Steven Emmerson
  • Re: [ldm-users] NEXRAD2/NEXRAD3 lag?
    • From: Blair Trosper
  • Re: [ldm-users] NEXRAD2/NEXRAD3 lag?
    • From: Karen Cooper - NOAA Affiliate