I'll have to take an alternate stance to both RAID5/6 and ext4. Our
experience with ext4 was not as good as ext2, but more than satisfactory.
However, xfs was the order of the day when I was still at A&M. Similarly,
the difference between RAID0 and RAID5 is noticable, but I had better than
"just satisfactory" performance using RAID6, and fewer worries about
hardware failures. I'm suspecting a time issue, offhand.
gerry
On Fri, May 23, 2014 at 1:13 PM, Robert Mullenax <
Robert.Mullenax@xxxxxxxxxxxxx> wrote:
> At this juncture it sure sounds like it almost has to be a disk
> performance issue. I went from HW RAID 0 to HW RAID 5 a few years ago as a
> test and was shocked to see how much slower writes are..not that I am
> advocating RAID 0. We use ZFS mirroring now.
>
>
>
> -----Original Message-----
> From: ldm-users-bounces@xxxxxxxxxxxxxxxx on behalf of Tyler Allison
> Sent: Fri 5/23/2014 12:53 PM
> To: Ryan Hickman
> Cc: LDM Users
> Subject: Re: [ldm-users] NEXRAD2/NEXRAD3 lag?
>
> I think you said you're running ext4 files system on the data directory?
>
> My testing of ext4 with high IO LDM servers was atrocious. We no longer
> use it for our high speed disk write areas.
>
> If you can, mount an ext2 partition and move your writes to there for a
> test .
>
> -Tyler
>
> > On May 23, 2014, at 12:34 PM, Ryan Hickman <ryan@xxxxxxxxxxxxxxxx>
> wrote:
> >
> > Let's see your NEXRAD2 pqact entries, please. We also didn't see the
> relevant 'ldmadmin watch' that Daryl suggested either so we could determine
> LDM product queue insertion time.
> >
> > This doesn't sound like a network topology or product dissemination
> issue; other list members would likely be seeing this problem, too.
> However, I'd be willing to add an "allow" for NEXRAD2 from AllisonHouse's
> Texas servers for 48 hours just to help you rule that out. Feel free to
> email me off list to accomplish this.
> >
> > Otherwise I don't have many other ideas.
> >
> > P.S. Daryl Herzmann has been trying to help. But it seems as if you
> aren't receiving his messages.
> http://www.unidata.ucar.edu/mailing_lists/archives/ldm-users/2014/msg00059.html
> >
> > Daryl - I noticed that you might be manually removing the email quotes
> in your replies. This MIGHT be throwing the Mailing List software off. If
> you trace the "Follow Ups" on Blair's original message (
> http://www.unidata.ucar.edu/mailing_lists/archives/ldm-users/2014/msg00044.html),
> I never see your replies. If you look at Blair's quoted messages, you'll
> see that his replies only quote the "Follow Ups" and they never include
> your replies.
> >
> > Blair - Consider adding Daryl to your email contacts. Or check your Spam
> / Junk folders for his previous messages.
> >
> >
> >> On Thu, May 22, 2014 at 8:29 PM, Blair Trosper <
> blair.trosper@xxxxxxxxxxxxxxxxxxx> wrote:
> >> We're still getting it from all of our servers, and I added another one
> in Denver with new providers again, and still seeing the same issue.
> Unique to NEXRAD2.
> >>
> >> Can anyone volunteer a server to feed us NEXRAD2 data for a few days
> (or as a backup permanently) to try and troubleshoot this?
> >>
> >> None of our servers is on the same network, so it cannot be that. And
> these are all rather powerful machines with extremely high connectivity and
> throughput...and we don't see latency on any of the other feeds we carry.
> >>
> >> We're only utilizing 1-2% of our total throughput on our Chicago
> server, for example, but we're getting every site 10 minutes delayed from:
> >> - OU
> >> - TAMU
> >> - WISC
> >>
> >> ...but none of us four shows any lag in the topology.
> >>
> >> What else could be going on?
> >>
> >>
> >>> On Sat, May 17, 2014 at 9:35 AM, Karen Cooper - NOAA Affiliate <
> karen.cooper@xxxxxxxx> wrote:
> >>> The level 2 data is inserted into Ldm queues at the radar RPGs as it
> is generated. If it were a problem there everyone would be seeing it.
> >>>
> >>> Sent from me! Please pardon any tpyos
> >>>
> >>>
> >>>> On May 17, 2014, at 1:18 AM, Blair Trosper <
> blair.trosper@xxxxxxxxxxxxxxxxxxx> wrote:
> >>>>
> >>>> No joy there. Our providers (TAMU, OU, WISC, et al) are all under
> tier 1 providers, and there is no lag.
> >>>>
> >>>> I can't find the document that describes how the NEXRAD2 data is fed
> into the LDM system. I think it happens in DC at the Internet2 aperture,
> but that information may be out of date (or I may be remembering it wrong).
> >>>>
> >>>> As I recall, two weeks ago, we weren't the only ones seeing this lag .
> >>>>
> >>>> And we're still seeing it on three servers with three separate
> upstream providers -- and each server is deliberately on different
> BGP/backbone providers for geographic AND network diversity.
> >>>>
> >>>> Something's not quite right, and I think the key is understanding how
> the NEXRAD2 data gets into LDM...and then going to THAT source to find out
> why it's delayed. I know Allison House's data isn't delayed (whereas
> iastate's is), so my guess is they're acquiring it independently of LDM.
> >>>>
> >>>> So we know iastate.edu is getting data 10 minutes late just like we
> are, as are our upstream providers...but the servers themselves are always
> at 0 or 1 second of lag.
> >>>>
> >>>> Something is definitely wrong. We don't see any lag on any of the
> other feed trees (HDS, IDS, WMO, NGRID, UNIWISC, NEXRAD3, etc). This seems
> to be unique to NEXRAD2.
> >>>>
> >>>> Can someone from UCAR/UNIDATA jump in and help us...or should we open
> a formal support ticket?
> >>>>
> >>>>
> >>>>> On Sun, May 11, 2014 at 5:49 PM, Steven Emmerson <
> emmerson.steven@xxxxxxxxx> wrote:
> >>>>> It could be that the last NEXRAD2 data-product is stuck in a system
> buffer. Try putting a "-flush" or "-close" on the relevant PIPE action.
> >>>>>
> >>>>>> On May 11, 2014 11:49 AM, "Gilbert Sebenste" <
> sebenste@xxxxxxxxxxxxxxxxxxxxx> wrote:
> >>>>>> Hi Blair,
> >>>>>>
> >>>>>>> We are still seeing this. I know the TOC was made aware of this
> issue two
> >>>>>>> weeks ago, but despite all our servers being 0 seconds lagged on
> the
> >>>>>>> NEXRAD2 feed tree, our data (from multiple providers who are also
> not
> >>>>>>> lagged) is at least 10 minutes behind.
> >>>>>>
> >>>>>> Been off the grid, for the most part, or I would have answered
> earlier. Things to look for:
> >>>>>>
> >>>>>> Can you do a traceroute to the servers in question? See if anything
> is
> >>>>>> jammed up inbetween them and you.
> >>>>>>
> >>>>>> Are all network cards set at least at 100 mb? If 10 mb, it can't
> handle it.
> >>>>>>
> >>>>>> Is the system clock correct? Are you using ntp? This is a must.
> >>>>>>
> >>>>>>> Also, this page is a 500 internal server error and has been for
> months, and nobody has fixed it or seems to even notice:
> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/rtstats_topogif?NEXRAD2
> >>>>>>>
> >>>>>>> And this one takes over a minute to load:
> >>>>>>>
> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/rtstats_feedtree?NEXRAD2
> >>>>>>>
> >>>>>>> ...but you can verify from the second link that we're not lagging
> (search
> >>>>>>> for "tampa" or "chicago").
> >>>>>>
> >>>>>> Give those a try aboove, and see what happens.
> >>>>>>
> >>>>>> Gilbert
> >>>>>>
> >>>>>>
> *******************************************************************************
> >>>>>> Gilbert Sebenste
> ********
> >>>>>> (My opinions
> only!) ******
> >>>>>> Staff Meteorologist, Northern Illinois
> University ****
> >>>>>> E-mail: sebenste@xxxxxxxxxxxxxxxxxxxxx
> ***
> >>>>>> web: http://weather.admin.niu.edu
> **
> >>>>>> Twitter: http://www.twitter.com/NIU_Weather
> **
> >>>>>> Facebook: http://www.facebook.com/niu.weather
> *
> >>>>>>
> *******************************************************************************
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> ldm-users mailing list
> >>>>>> ldm-users@xxxxxxxxxxxxxxxx
> >>>>>> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
> >>>>>
> >>>>> _______________________________________________
> >>>>> ldm-users mailing list
> >>>>> ldm-users@xxxxxxxxxxxxxxxx
> >>>>> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>>
> >>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
>
> >>>> Blair Trosper
> >>>> Updraft Networks / Weather Data
> >>>> NOC: 469-844-5440
> >>>> Early Watch Notifications: http://twitter.com/weatherwatches
> >>>> _______________________________________________
> >>>> ldm-users mailing list
> >>>> ldm-users@xxxxxxxxxxxxxxxx
> >>>> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
> >>
> >>
> >>
> >> --
> >>
> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
>
> >> Blair Trosper
> >> Updraft Networks / Weather Data
> >> NOC: 469-844-5440
> >> Early Watch Notifications: http://twitter.com/weatherwatches
> >>
> >> _______________________________________________
> >> ldm-users mailing list
> >> ldm-users@xxxxxxxxxxxxxxxx
> >> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
> >
> > _______________________________________________
> > ldm-users mailing list
> > ldm-users@xxxxxxxxxxxxxxxx
> > For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
>
> _______________________________________________
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
--
Gerry Creager
NSSL/CIMMS
405.325.6371
++++++++++++++++++++++
“Big whorls have little whorls,
That feed on their velocity;
And little whorls have lesser whorls,
And so on to viscosity.”
Lewis Fry Richardson (1881-1953)
