And it's ext4.
This is a new problem...it started about three weeks ago. For the two
years prior to that, we saw NO delays *ever*. And, as I mentioned, others
like iastate also have observed this, as have our upstream providers (who
are getting their data from the tier 1 servers).
On Sat, May 17, 2014 at 4:06 AM, Blair Trosper <
blair.trosper@xxxxxxxxxxxxxxxxxxx> wrote:
> And it's Debian 7
>
>
> On Sat, May 17, 2014 at 4:06 AM, Blair Trosper <
> blair.trosper@xxxxxxxxxxxxxxxxxxx> wrote:
>
>> Yes. SAS drives on all three machines, two are Intel Xeon Ivy
>> Bridge...the other is an AMD hexacore running at 3.6 GHz. Each box has
>> 1Gbps symmetric and at least 16GB RAM.
>>
>>
>> On Sat, May 17, 2014 at 3:33 AM, Robert Mullenax <
>> Robert.Mullenax@xxxxxxxxxxxxx> wrote:
>>
>>> Are you 100% sure it isn't a local write to disk issue..not a big enough
>>> queue or general performance issues. What OS, file system, and type of
>>> disks do you have?
>>>
>>>
>>>
>>> On May 17, 2014, at 12:45 AM, "Blair Trosper" <
>>> blair.trosper@xxxxxxxxxxxxxxxxxxx> wrote:
>>>
>>> Odd. We watch the Internet2 weathermap in real time when we see delay,
>>> and none of the 10Gbps connections are ever more than 1% utilized. So it's
>>> not a problem inside Internet2.
>>>
>>> Can someone help me remember where the document is that explains how
>>> data gets from NOAA into the LDM feed trees?
>>>
>>> The last one I saw was a few years ago, and I absolutely cannot remember
>>> how I found it. Unfortunately, as with a lot of things on the UNIDATA site
>>> (like rtstats), there are often broken links or server errors -- or
>>> documents that are just flat our wrong or out of date as well.
>>>
>>>
>>> On Sat, May 17, 2014 at 1:38 AM, Ryan Hickman <ryan@xxxxxxxxxxxxxxxx>wrote:
>>>
>>>> We (AllisonHouse) do not acquire the NEXRAD2 data outside of LDM. In
>>>> fact, we share common upstream providers.
>>>>
>>>> This sounds like an issue that is unique to Internet2.
>>>>
>>>> - Ryan Hickman
>>>> Chief Technology Officer
>>>> AllisonHouse
>>>>
>>>> On May 17, 2014, at 12:18 AM, Blair Trosper <
>>>> blair.trosper@xxxxxxxxxxxxxxxxxxx> wrote:
>>>>
>>>> No joy there. Our providers (TAMU, OU, WISC, et al) are all under tier
>>>> 1 providers, and there is no lag.
>>>>
>>>> I can't find the document that describes how the NEXRAD2 data is fed
>>>> into the LDM system. I think it happens in DC at the Internet2 aperture,
>>>> but that information may be out of date (or I may be remembering it wrong).
>>>>
>>>> As I recall, two weeks ago, we weren't the only ones seeing this lag.
>>>>
>>>> And we're still seeing it on three servers with three separate upstream
>>>> providers -- and each server is deliberately on different BGP/backbone
>>>> providers for geographic AND network diversity.
>>>>
>>>> Something's not quite right, and I think the key is understanding how
>>>> the NEXRAD2 data gets into LDM...and then going to THAT source to find out
>>>> why it's delayed. I know Allison House's data isn't delayed (whereas
>>>> iastate's is), so my guess is they're acquiring it independently of LDM.
>>>>
>>>> So we know iastate.edu is getting data 10 minutes late just like we
>>>> are, as are our upstream providers...but the servers themselves are always
>>>> at 0 or 1 second of lag.
>>>>
>>>> Something is definitely wrong. We don't see any lag on any of the
>>>> other feed trees (HDS, IDS, WMO, NGRID, UNIWISC, NEXRAD3, etc). This seems
>>>> to be unique to NEXRAD2.
>>>>
>>>> Can someone from UCAR/UNIDATA jump in and help us...or should we open a
>>>> formal support ticket?
>>>>
>>>>
>>>> On Sun, May 11, 2014 at 5:49 PM, Steven Emmerson <
>>>> emmerson.steven@xxxxxxxxx> wrote:
>>>>
>>>>> It could be that the last NEXRAD2 data-product is stuck in a system
>>>>> buffer. Try putting a "-flush" or "-close" on the relevant PIPE action.
>>>>> On May 11, 2014 11:49 AM, "Gilbert Sebenste" <
>>>>> sebenste@xxxxxxxxxxxxxxxxxxxxx> wrote:
>>>>>
>>>>>> Hi Blair,
>>>>>>
>>>>>> We are still seeing this. I know the TOC was made aware of this
>>>>>>> issue two
>>>>>>> weeks ago, but despite all our servers being 0 seconds lagged on the
>>>>>>> NEXRAD2 feed tree, our data (from multiple providers who are also not
>>>>>>> lagged) is at least 10 minutes behind.
>>>>>>>
>>>>>>
>>>>>> Been off the grid, for the most part, or I would have answered
>>>>>> earlier. Things to look for:
>>>>>>
>>>>>> Can you do a traceroute to the servers in question? See if anything is
>>>>>> jammed up inbetween them and you.
>>>>>>
>>>>>> Are all network cards set at least at 100 mb? If 10 mb, it can't
>>>>>> handle it.
>>>>>>
>>>>>> Is the system clock correct? Are you using ntp? This is a must.
>>>>>>
>>>>>> Also, this page is a 500 internal server error and has been for
>>>>>>> months, and nobody has fixed it or seems to even notice:
>>>>>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/rtstats_
>>>>>>> topogif?NEXRAD2
>>>>>>>
>>>>>>> And this one takes over a minute to load:
>>>>>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/rtstats_
>>>>>>> feedtree?NEXRAD2
>>>>>>>
>>>>>>> ...but you can verify from the second link that we're not lagging
>>>>>>> (search
>>>>>>> for "tampa" or "chicago").
>>>>>>>
>>>>>>
>>>>>> Give those a try aboove, and see what happens.
>>>>>>
>>>>>> Gilbert
>>>>>>
>>>>>> ************************************************************
>>>>>> *******************
>>>>>> Gilbert Sebenste
>>>>>> ********
>>>>>> (My opinions only!)
>>>>>> ******
>>>>>> Staff Meteorologist, Northern Illinois University
>>>>>> ****
>>>>>> E-mail: sebenste@xxxxxxxxxxxxxxxxxxxxx
>>>>>> ***
>>>>>> web: http://weather.admin.niu.edu
>>>>>> **
>>>>>> Twitter: http://www.twitter.com/NIU_Weather
>>>>>> **
>>>>>> Facebook: http://www.facebook.com/niu.weather
>>>>>> *
>>>>>> ************************************************************
>>>>>> *******************
>>>>>>
>>>>>> _______________________________________________
>>>>>> ldm-users mailing list
>>>>>> ldm-users@xxxxxxxxxxxxxxxx
>>>>>> For list information or to unsubscribe, visit:
>>>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> ldm-users mailing list
>>>>> ldm-users@xxxxxxxxxxxxxxxx
>>>>> For list information or to unsubscribe, visit:
>>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙
>>>> ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙
>>>> ∙
>>>> Blair Trosper
>>>> Updraft Networks / Weather Data
>>>> NOC: 469-844-5440
>>>> Early Watch Notifications: http://twitter.com/weatherwatches
>>>>
>>>> _______________________________________________
>>>> ldm-users mailing list
>>>> ldm-users@xxxxxxxxxxxxxxxx
>>>> For list information or to unsubscribe, visit:
>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙
>>> ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙
>>> Blair Trosper
>>> Updraft Networks / Weather Data
>>> NOC: 469-844-5440
>>> Early Watch Notifications: http://twitter.com/weatherwatches
>>>
>>> _______________________________________________
>>> ldm-users mailing list
>>> ldm-users@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe, visit:
>>> http://www.unidata.ucar.edu/mailing_lists/
>>>
>>>
>>
>>
>> --
>>
>> ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙
>> ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙
>> Blair Trosper
>> Updraft Networks / Weather Data
>> NOC: 469-844-5440
>> Early Watch Notifications: http://twitter.com/weatherwatches
>>
>
>
>
> --
>
> ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙
> ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙
> Blair Trosper
> Updraft Networks / Weather Data
> NOC: 469-844-5440
> Early Watch Notifications: http://twitter.com/weatherwatches
>
--
∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙
∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙
Blair Trosper
Updraft Networks / Weather Data
NOC: 469-844-5440
Early Watch Notifications: http://twitter.com/weatherwatches
