At this juncture it sure sounds like it almost has to be a disk performance
issue. I went from HW RAID 0 to HW RAID 5 a few years ago as a test and was
shocked to see how much slower writes are..not that I am advocating RAID 0. We
use ZFS mirroring now.
-----Original Message-----
From: ldm-users-bounces@xxxxxxxxxxxxxxxx on behalf of Tyler Allison
Sent: Fri 5/23/2014 12:53 PM
To: Ryan Hickman
Cc: LDM Users
Subject: Re: [ldm-users] NEXRAD2/NEXRAD3 lag?
I think you said you're running ext4 files system on the data directory?
My testing of ext4 with high IO LDM servers was atrocious. We no longer use it
for our high speed disk write areas.
If you can, mount an ext2 partition and move your writes to there for a test .
-Tyler
> On May 23, 2014, at 12:34 PM, Ryan Hickman <ryan@xxxxxxxxxxxxxxxx> wrote:
>
> Let's see your NEXRAD2 pqact entries, please. We also didn't see the relevant
> 'ldmadmin watch' that Daryl suggested either so we could determine LDM
> product queue insertion time.
>
> This doesn't sound like a network topology or product dissemination issue;
> other list members would likely be seeing this problem, too. However, I'd be
> willing to add an "allow" for NEXRAD2 from AllisonHouse's Texas servers for
> 48 hours just to help you rule that out. Feel free to email me off list to
> accomplish this.
>
> Otherwise I don't have many other ideas.
>
> P.S. Daryl Herzmann has been trying to help. But it seems as if you aren't
> receiving his messages.
> http://www.unidata.ucar.edu/mailing_lists/archives/ldm-users/2014/msg00059.html
>
> Daryl - I noticed that you might be manually removing the email quotes in
> your replies. This MIGHT be throwing the Mailing List software off. If you
> trace the "Follow Ups" on Blair's original message
> (http://www.unidata.ucar.edu/mailing_lists/archives/ldm-users/2014/msg00044.html),
> I never see your replies. If you look at Blair's quoted messages, you'll see
> that his replies only quote the "Follow Ups" and they never include your
> replies.
>
> Blair - Consider adding Daryl to your email contacts. Or check your Spam /
> Junk folders for his previous messages.
>
>
>> On Thu, May 22, 2014 at 8:29 PM, Blair Trosper
>> <blair.trosper@xxxxxxxxxxxxxxxxxxx> wrote:
>> We're still getting it from all of our servers, and I added another one in
>> Denver with new providers again, and still seeing the same issue. Unique
>> to NEXRAD2.
>>
>> Can anyone volunteer a server to feed us NEXRAD2 data for a few days (or as
>> a backup permanently) to try and troubleshoot this?
>>
>> None of our servers is on the same network, so it cannot be that. And these
>> are all rather powerful machines with extremely high connectivity and
>> throughput...and we don't see latency on any of the other feeds we carry.
>>
>> We're only utilizing 1-2% of our total throughput on our Chicago server, for
>> example, but we're getting every site 10 minutes delayed from:
>> - OU
>> - TAMU
>> - WISC
>>
>> ...but none of us four shows any lag in the topology.
>>
>> What else could be going on?
>>
>>
>>> On Sat, May 17, 2014 at 9:35 AM, Karen Cooper - NOAA Affiliate
>>> <karen.cooper@xxxxxxxx> wrote:
>>> The level 2 data is inserted into Ldm queues at the radar RPGs as it is
>>> generated. If it were a problem there everyone would be seeing it.
>>>
>>> Sent from me! Please pardon any tpyos
>>>
>>>
>>>> On May 17, 2014, at 1:18 AM, Blair Trosper
>>>> <blair.trosper@xxxxxxxxxxxxxxxxxxx> wrote:
>>>>
>>>> No joy there. Our providers (TAMU, OU, WISC, et al) are all under tier 1
>>>> providers, and there is no lag.
>>>>
>>>> I can't find the document that describes how the NEXRAD2 data is fed into
>>>> the LDM system. I think it happens in DC at the Internet2 aperture, but
>>>> that information may be out of date (or I may be remembering it wrong).
>>>>
>>>> As I recall, two weeks ago, we weren't the only ones seeing this lag.
>>>>
>>>> And we're still seeing it on three servers with three separate upstream
>>>> providers -- and each server is deliberately on different BGP/backbone
>>>> providers for geographic AND network diversity.
>>>>
>>>> Something's not quite right, and I think the key is understanding how the
>>>> NEXRAD2 data gets into LDM...and then going to THAT source to find out why
>>>> it's delayed. I know Allison House's data isn't delayed (whereas
>>>> iastate's is), so my guess is they're acquiring it independently of LDM.
>>>>
>>>> So we know iastate.edu is getting data 10 minutes late just like we are,
>>>> as are our upstream providers...but the servers themselves are always at 0
>>>> or 1 second of lag.
>>>>
>>>> Something is definitely wrong. We don't see any lag on any of the other
>>>> feed trees (HDS, IDS, WMO, NGRID, UNIWISC, NEXRAD3, etc). This seems to
>>>> be unique to NEXRAD2.
>>>>
>>>> Can someone from UCAR/UNIDATA jump in and help us...or should we open a
>>>> formal support ticket?
>>>>
>>>>
>>>>> On Sun, May 11, 2014 at 5:49 PM, Steven Emmerson
>>>>> <emmerson.steven@xxxxxxxxx> wrote:
>>>>> It could be that the last NEXRAD2 data-product is stuck in a system
>>>>> buffer. Try putting a "-flush" or "-close" on the relevant PIPE action.
>>>>>
>>>>>> On May 11, 2014 11:49 AM, "Gilbert Sebenste"
>>>>>> <sebenste@xxxxxxxxxxxxxxxxxxxxx> wrote:
>>>>>> Hi Blair,
>>>>>>
>>>>>>> We are still seeing this. I know the TOC was made aware of this issue
>>>>>>> two
>>>>>>> weeks ago, but despite all our servers being 0 seconds lagged on the
>>>>>>> NEXRAD2 feed tree, our data (from multiple providers who are also not
>>>>>>> lagged) is at least 10 minutes behind.
>>>>>>
>>>>>> Been off the grid, for the most part, or I would have answered earlier.
>>>>>> Things to look for:
>>>>>>
>>>>>> Can you do a traceroute to the servers in question? See if anything is
>>>>>> jammed up inbetween them and you.
>>>>>>
>>>>>> Are all network cards set at least at 100 mb? If 10 mb, it can't handle
>>>>>> it.
>>>>>>
>>>>>> Is the system clock correct? Are you using ntp? This is a must.
>>>>>>
>>>>>>> Also, this page is a 500 internal server error and has been for months,
>>>>>>> and nobody has fixed it or seems to even notice:
>>>>>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/rtstats_topogif?NEXRAD2
>>>>>>>
>>>>>>> And this one takes over a minute to load:
>>>>>>> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/rtstats_feedtree?NEXRAD2
>>>>>>>
>>>>>>> ...but you can verify from the second link that we're not lagging
>>>>>>> (search
>>>>>>> for "tampa" or "chicago").
>>>>>>
>>>>>> Give those a try aboove, and see what happens.
>>>>>>
>>>>>> Gilbert
>>>>>>
>>>>>> *******************************************************************************
>>>>>> Gilbert Sebenste
>>>>>> ********
>>>>>> (My opinions only!)
>>>>>> ******
>>>>>> Staff Meteorologist, Northern Illinois University
>>>>>> ****
>>>>>> E-mail: sebenste@xxxxxxxxxxxxxxxxxxxxx
>>>>>> ***
>>>>>> web: http://weather.admin.niu.edu **
>>>>>> Twitter: http://www.twitter.com/NIU_Weather **
>>>>>> Facebook: http://www.facebook.com/niu.weather *
>>>>>> *******************************************************************************
>>>>>>
>>>>>> _______________________________________________
>>>>>> ldm-users mailing list
>>>>>> ldm-users@xxxxxxxxxxxxxxxx
>>>>>> For list information or to unsubscribe, visit:
>>>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>>
>>>>> _______________________________________________
>>>>> ldm-users mailing list
>>>>> ldm-users@xxxxxxxxxxxxxxxx
>>>>> For list information or to unsubscribe, visit:
>>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
>>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
>>>> Blair Trosper
>>>> Updraft Networks / Weather Data
>>>> NOC: 469-844-5440
>>>> Early Watch Notifications: http://twitter.com/weatherwatches
>>>> _______________________________________________
>>>> ldm-users mailing list
>>>> ldm-users@xxxxxxxxxxxxxxxx
>>>> For list information or to unsubscribe, visit:
>>>> http://www.unidata.ucar.edu/mailing_lists/
>>
>>
>>
>> --
>>
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
>> Blair Trosper
>> Updraft Networks / Weather Data
>> NOC: 469-844-5440
>> Early Watch Notifications: http://twitter.com/weatherwatches
>>
>> _______________________________________________
>> ldm-users mailing list
>> ldm-users@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>
> _______________________________________________
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
