I’ll second Ryan’s suggestion: Please post a sample output of “ldmadmin watch”
From: ldm-users-bounces@xxxxxxxxxxxxxxxx
[mailto:ldm-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Ryan Hickman
Sent: Friday, May 23, 2014 12:34
To: Blair Trosper
Cc: LDM Users
Subject: Re: [ldm-users] NEXRAD2/NEXRAD3 lag?
Let's see your NEXRAD2 pqact entries, please. We also didn't see the relevant
'ldmadmin watch' that Daryl suggested either so we could determine LDM product
queue insertion time.
This doesn't sound like a network topology or product dissemination issue;
other list members would likely be seeing this problem, too. However, I'd be
willing to add an "allow" for NEXRAD2 from AllisonHouse's Texas servers for 48
hours just to help you rule that out. Feel free to email me off list to
accomplish this.
Otherwise I don't have many other ideas.
P.S. Daryl Herzmann has been trying to help. But it seems as if you aren't
receiving his messages.
http://www.unidata.ucar.edu/mailing_lists/archives/ldm-users/2014/msg00059.html
Daryl - I noticed that you might be manually removing the email quotes in your
replies. This MIGHT be throwing the Mailing List software off. If you trace the
"Follow Ups" on Blair's original message
(http://www.unidata.ucar.edu/mailing_lists/archives/ldm-users/2014/msg00044.html),
I never see your replies. If you look at Blair's quoted messages, you'll see
that his replies only quote the "Follow Ups" and they never include your
replies.
Blair - Consider adding Daryl to your email contacts. Or check your Spam / Junk
folders for his previous messages.
On Thu, May 22, 2014 at 8:29 PM, Blair Trosper
<blair.trosper@xxxxxxxxxxxxxxxxxxx<mailto:blair.trosper@xxxxxxxxxxxxxxxxxxx>>
wrote:
We're still getting it from all of our servers, and I added another one in
Denver with new providers again, and still seeing the same issue. Unique to
NEXRAD2.
Can anyone volunteer a server to feed us NEXRAD2 data for a few days (or as a
backup permanently) to try and troubleshoot this?
None of our servers is on the same network, so it cannot be that. And these
are all rather powerful machines with extremely high connectivity and
throughput...and we don't see latency on any of the other feeds we carry.
We're only utilizing 1-2% of our total throughput on our Chicago server, for
example, but we're getting every site 10 minutes delayed from:
- OU
- TAMU
- WISC
...but none of us four shows any lag in the topology.
What else could be going on?
On Sat, May 17, 2014 at 9:35 AM, Karen Cooper - NOAA Affiliate
<karen.cooper@xxxxxxxx<mailto:karen.cooper@xxxxxxxx>> wrote:
The level 2 data is inserted into Ldm queues at the radar RPGs as it is
generated. If it were a problem there everyone would be seeing it.
Sent from me! Please pardon any tpyos
On May 17, 2014, at 1:18 AM, Blair Trosper
<blair.trosper@xxxxxxxxxxxxxxxxxxx<mailto:blair.trosper@xxxxxxxxxxxxxxxxxxx>>
wrote:
No joy there. Our providers (TAMU, OU, WISC, et al) are all under tier 1
providers, and there is no lag.
I can't find the document that describes how the NEXRAD2 data is fed into the
LDM system. I think it happens in DC at the Internet2 aperture, but that
information may be out of date (or I may be remembering it wrong).
As I recall, two weeks ago, we weren't the only ones seeing this lag.
And we're still seeing it on three servers with three separate upstream
providers -- and each server is deliberately on different BGP/backbone
providers for geographic AND network diversity.
Something's not quite right, and I think the key is understanding how the
NEXRAD2 data gets into LDM...and then going to THAT source to find out why it's
delayed. I know Allison House's data isn't delayed (whereas iastate's is), so
my guess is they're acquiring it independently of LDM.
So we know iastate.edu<http://iastate.edu> is getting data 10 minutes late just
like we are, as are our upstream providers...but the servers themselves are
always at 0 or 1 second of lag.
Something is definitely wrong. We don't see any lag on any of the other feed
trees (HDS, IDS, WMO, NGRID, UNIWISC, NEXRAD3, etc). This seems to be unique
to NEXRAD2.
Can someone from UCAR/UNIDATA jump in and help us...or should we open a formal
support ticket?
On Sun, May 11, 2014 at 5:49 PM, Steven Emmerson
<emmerson.steven@xxxxxxxxx<mailto:emmerson.steven@xxxxxxxxx>> wrote:
It could be that the last NEXRAD2 data-product is stuck in a system buffer. Try
putting a "-flush" or "-close" on the relevant PIPE action.
On May 11, 2014 11:49 AM, "Gilbert Sebenste"
<sebenste@xxxxxxxxxxxxxxxxxxxxx<mailto:sebenste@xxxxxxxxxxxxxxxxxxxxx>> wrote:
Hi Blair,
We are still seeing this. I know the TOC was made aware of this issue two
weeks ago, but despite all our servers being 0 seconds lagged on the
NEXRAD2 feed tree, our data (from multiple providers who are also not
lagged) is at least 10 minutes behind.
Been off the grid, for the most part, or I would have answered earlier. Things
to look for:
Can you do a traceroute to the servers in question? See if anything is
jammed up inbetween them and you.
Are all network cards set at least at 100 mb? If 10 mb, it can't handle it.
Is the system clock correct? Are you using ntp? This is a must.
Also, this page is a 500 internal server error and has been for months, and
nobody has fixed it or seems to even notice:
http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/rtstats_topogif?NEXRAD2
And this one takes over a minute to load:
http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/rtstats_feedtree?NEXRAD2
...but you can verify from the second link that we're not lagging (search
for "tampa" or "chicago").
Give those a try aboove, and see what happens.
Gilbert
*******************************************************************************
Gilbert Sebenste ********
(My opinions only!) ******
Staff Meteorologist, Northern Illinois University ****
E-mail: sebenste@xxxxxxxxxxxxxxxxxxxxx<mailto:sebenste@xxxxxxxxxxxxxxxxxxxxx>
***
web: http://weather.admin.niu.edu **
Twitter: http://www.twitter.com/NIU_Weather **
Facebook: http://www.facebook.com/niu.weather *
*******************************************************************************
_______________________________________________
ldm-users mailing list
ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
_______________________________________________
ldm-users mailing list
ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
--
∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙
∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙
Blair Trosper
Updraft Networks / Weather Data
NOC: 469-844-5440<tel:469-844-5440>
Early Watch Notifications: http://twitter.com/weatherwatches
_______________________________________________
ldm-users mailing list
ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
--
∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙
∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙ ∙
Blair Trosper
Updraft Networks / Weather Data
NOC: 469-844-5440<tel:469-844-5440>
Early Watch Notifications: http://twitter.com/weatherwatches
_______________________________________________
ldm-users mailing list
ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
