[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: LDM blues ... was: Re[2]: IDD latencies



Chris Herbster wrote:
> 
> Anne et al.,
> 
> Thanks for the offer to help.  I too would prefer to solve this myself with 
> your
> help, but I just can't find the time.
> 
> To make things even more interesting, my wife and I had a son back in June!  I
> think that satisfies the question on whether I have something to do with my
> "free time!"  (-:
> 
> As for how fast we fix this, well, I've been trying to get to it for quite a 
> few
> months, and our only plan for a feed out of us is to our Prescott campus ....
> 
> The current ldm box is:  thermal.db.erau.edu
> 
> Name:    thermal.db.erau.edu
> Address:  155.31.129.19
> 
> We're running SSH2.  Please let me know if you have any problems....
> 
> "ldm" is in the sudoers list to sudo all commands.
> 
> I can't thank you enough.  I'm willing to bet that you'll find some basic
> problems that I have not been able to get past ....
> 
> Could you please create a log file of the things that you do so I can follow
> your trail?  This way, you'll still be able to teach me about what is wrong.
> 
> Yes, when we get this working than I'd love to be able to just copy everything
> over to the new machine ("updraft").  We might want to adjust the whole data
> tree to that disk, as there is more free space on that NFS mount than is local
> to thermal.
> 
> I'll check to see if I'm current with GEMPAK and MCIDAS.  I think I might be a
> minor release behind in GEMPAK ....
> 
> Once again, I can't thank you enough!
> 
> Cheers,
> 
> Chris H.
> 

Hi Chris,

Well, here's the trail of what I did and my conclusions at each point:

First, I did 'ldmadmin watch' to see what's coming in and how timely it
is.  Right away I can see there's a problem:

[ldm@thermal ~]$ ldmadmin watch
(Type ^D or ^C when finished)
Oct 02 21:55:35 pqutil:    15112 20011002205508.163 IDS|DDPLUS 421 
FNUS54 KTSA 022055 /pFWFTUL
Oct 02 21:55:35 pqutil:      132 20011002205508.194 IDS|DDPLUS 423 
SAUS42 KMLB 022054 /pMTRMCO
Oct 02 21:55:36 pqutil:      139 20011002205508.197 IDS|DDPLUS 424 
SAUS45 KABQ 022054 /pMTRGNT
Oct 02 21:55:42 pqutil:    10564 20011002205510.645 IDS|DDPLUS 431 
CXUS86 KSCS 022000 /pRSDMT
Oct 02 21:55:42 pqutil:      872 20011002205510.680 IDS|DDPLUS 432 
CXUS86 KSCS 022000 /pRSDNM
Oct 02 21:55:43 pqutil:      158 20011002205510.760 IDS|DDPLUS 435 
SAUS46 KLOX 022054 /pMTRLAX
Oct 02 21:55:43 pqutil:      151 20011002205510.762 IDS|DDPLUS 436 
FNUS84 KSHV 022055 /pFWMSHV
Oct 02 21:55:45 pqutil:      132 20011002205510.764 IDS|DDPLUS 437 
SAUS43 KMKX 022055 /pMTRMSN
Oct 02 21:55:45 pqutil:      129 20011002205510.776 IDS|DDPLUS 438 
SAUS44 KMRX 022054 /pMTRTRI
Oct 02 21:55:46 pqutil:     1558 20011002205510.776 IDS|DDPLUS 439 
AWUS84 KSHV 022055 /pAWSSHV
Oct 02 21:55:47 pqutil:      129 20011002205510.777 IDS|DDPLUS 440 
SAUS42 KFFC 022054 /pMTRMCN
Oct 02 21:55:47 pqutil:      135 20011002205510.777 IDS|DDPLUS 441 
SAUS45 KRIW 022054 /pMTRLND
Oct 02 21:55:47 pqutil:      141 20011002205510.778 IDS|DDPLUS 442 
SAUS46 KSEW 022054 /pMTRUIL
Oct 02 21:55:48 pqutil:      145 20011002205510.778 IDS|DDPLUS 443 
SAUS44 KEWX 022054 /pMTRDRT
Oct 02 21:55:51 pqutil:     2787 20011002205510.779 IDS|DDPLUS 444 
CXUS86 KSCS 022000 /pRSDNV
Oct 02 21:55:51 pqutil:      129 20011002205510.779 IDS|DDPLUS 445 
SAUS44 KBMX 022054 /pMTRMGM
Oct 02 21:55:52 pqutil:      139 20011002205510.780 IDS|DDPLUS 446 
SAUS44 KHGX 022054 /pMTRIAH
Oct 02 21:55:52 pqutil:      123 20011002205510.780 IDS|DDPLUS 447 
SXUS43 KGLD 022054 /pMTTSYF
Oct 02 21:55:53 pqutil:      142 20011002205510.780 IDS|DDPLUS 448 
SAUS45 KABQ 022055 /pMTRRTN
Oct 02 21:55:58 pqutil:     8587 20011002205512.890 IDS|DDPLUS 462 
CXUS86 KSCS 022000 /pRSDOR
Oct 02 21:56:00 pqutil:      566 20011002205512.896 IDS|DDPLUS 463 
CXUS86 KSCS 022000 /pRSDSD
Oct 02 21:56:01 pqutil:      129 20011002205517.154 IDS|DDPLUS 512 
SAUS44 KLIX 022055 /pMTRBTR

If you take the difference between the current time, field 3, and the
ingest time, field 6, you'll see that you are receiving products that
over an hour old.  By default, these products will be rejected by your
ldm.  To confirm this, I took a look in your logs.  Here's what I found:

Oct 02 04:01:35 thermal pluto[25712]: RECLASS: 20011002030135.829
TS_ENDT {{DDPLUS,  ".*"},{HDS,  ".*"}} 
Oct 02 04:01:35 thermal pluto[25712]: skipped: 20011002030035.679
(60.150 seconds) 
Oct 02 04:02:58 thermal pluto[25712]: RECLASS: 20011002030258.792
TS_ENDT {{DDPLUS,  ".*"},{HDS,  ".*"}} 
Oct 02 04:02:58 thermal pluto[25712]: skipped: 20011002030158.322
(60.471 seconds) 
Oct 02 04:04:36 thermal pluto[25712]: RECLASS: 20011002030436.809
TS_ENDT {{DDPLUS,  ".*"},{HDS,  ".*"}} 
Oct 02 04:04:36 thermal pluto[25712]: skipped: 20011002030334.886
(61.923 seconds) 
Oct 02 04:06:36 thermal pluto[25712]: RECLASS: 20011002030636.111
TS_ENDT {{DDPLUS,  ".*"},{HDS,  ".*"}} 
Oct 02 04:06:36 thermal pluto[25712]: skipped: 20011002030535.628
(60.483 seconds) 
Oct 02 04:09:05 thermal pluto[25712]: RECLASS: 20011002030905.860
TS_ENDT {{DDPLUS,  ".*"},{HDS,  ".*"}} 
Oct 02 04:09:05 thermal pluto[25712]: skipped: 20011002030805.514
(60.346 seconds) 
Oct 02 04:11:27 thermal pluto[25712]: RECLASS: 20011002031127.262
TS_ENDT {{DDPLUS,  ".*"},{HDS,  ".*"}} 
Oct 02 04:11:27 thermal pluto[25712]: skipped: 20011002031026.513
(60.749 seconds) 
Oct 02 04:14:53 thermal pluto[25712]: RECLASS: 20011002031453.482
TS_ENDT {{DDPLUS,  ".*"},{HDS,  ".*"}} 
Oct 02 04:14:53 thermal pluto[25712]: skipped: 20011002031353.014
(60.469 seconds) 
Oct 02 04:25:17 thermal pluto[25712]: RECLASS: 20011002032517.249
TS_ENDT {{DDPLUS,  ".*"},{HDS,  ".*"}} 
Oct 02 04:25:17 thermal pluto[25712]: skipped: 20011002032415.947
(61.302 seconds) 
Oct 02 04:30:17 thermal pluto[25712]: RECLASS: 20011002033017.719
TS_ENDT {{DDPLUS,  ".*"},{HDS,  ".*"}} 
Oct 02 04:30:17 thermal pluto[25712]: skipped: 20011002032917.665
(60.054 seconds) 
Oct 02 04:44:12 thermal pluto[25712]: RECLASS: 20011002034412.468
TS_ENDT {{DDPLUS,  ".*"},{HDS,  ".*"}} 
Oct 02 04:44:12 thermal pluto[25712]: skipped: 20011002034312.417
(60.051 seconds) 

The 'skipped' lines mean that your ldm rejected those products.  They
also show that each product was just 60 seconds over the time limit,
which is one hour, be default.  Looks like this has been going on for a
while.

I decided to check your system clock to make sure it was running on
time.  I compared it with my own.  Mine is synchronized via NNTP, so
it's pretty accurate.  Here's what I got:

[ldm@thermal ~/logs]$ date -u
Tue Oct  2 21:56:40 UTC 2001
(anne) imogene:/home/anne/conferences/AMS 59 % date -u
Tue Oct  2 21:53:34 UTC 2001

So your clock is ahead by three minutes.  Generally that wouldn't be a
big deal, but it looks like you're running on the edge, with products
being rejected for being 60+ seconds late.  Resetting your clock would
help.

My next question is to figure out why there's such a delay.  From your
ldmd.conf I see that your primary upstream feed is pluto.met.fsu.edu. 
I'd like to know if they're getting products on time.  So, I did a
notifyme to them from your machine:

[ldm@thermal ~/etc]$ notifyme -vl - -h pluto.met.fsu.edu
Oct 02 22:08:39 notifyme[3037]: Starting Up: pluto.met.fsu.edu:
20011002220839.111 TS_ENDT {{ANY,  ".*"}}
Oct 02 22:08:39 notifyme[3037]: NOTIFYME(pluto.met.fsu.edu): reclass:
20011002220839.111 TS_ENDT {{FSL2|UNIDATA,  ".*"}}
Oct 02 22:08:40 notifyme[3037]: NOTIFYME(pluto.met.fsu.edu): OK
...

And then I waited a long time.  For comparison, I did a notifyme to them
from my own machine:


(anne) imogene:/home/anne/conferences/AMS 60 % notifyme -vl - -h
pluto.met.fsu.edu
Oct 02 22:06:18 notifyme[8932]: Starting Up: pluto.met.fsu.edu:
20011002220618.206 TS_ENDT {{ANY,  ".*"}}
Oct 02 22:06:18 notifyme[8932]: NOTIFYME(pluto.met.fsu.edu): OK
ct 02 22:07:23 notifyme[8932]:     1344 20011002220654.903    NLDN 000 
2001275220005
Oct 02 22:07:25 notifyme[8932]:    13050 20011002220723.458     WSI
21787  NEX/YUX/BREF2/200110022200
Oct 02 22:07:26 notifyme[8932]:    14838 20011002220725.288     WSI
21788  NEX/EMX/VEL1/200110022205
Oct 02 22:07:28 notifyme[8932]:    16770 20011002220726.315     WSI
21789  NEX/EMX/SRMV1/200110022205
Oct 02 22:07:28 notifyme[8932]:    12184 20011002220728.159     WSI
21790  NEX/EMX/BREF2/200110022205
Oct 02 22:07:44 notifyme[8932]:     1025 20011002220618.449 IDS|DDPLUS
625  SXUS70 KWAL 022203
Oct 02 22:07:44 notifyme[8932]:      301 20011002220618.452 IDS|DDPLUS
626  SXUS70 KWAL 022203
Oct 02 22:07:44 notifyme[8932]:      300 20011002220618.455 IDS|DDPLUS
627  SXUS70 KWAL 022203
Oct 02 22:07:44 notifyme[8932]:     8153 20011002220743.887 NNEXRAD 783 
SDUS52 KFFC 022206 /pN0RJGX
Oct 02 22:07:45 notifyme[8932]:      307 20011002220618.458 IDS|DDPLUS
628  SXUS70 KWAL 022203
Oct 02 22:07:45 notifyme[8932]:     1485 20011002220618.463 IDS|DDPLUS
629  SRUS81 KALY 022205 /pRRAALB
Oct 02 22:07:46 notifyme[8932]:      300 20011002220618.467 IDS|DDPLUS
630  SXUS70 KWAL 022203
Oct 02 22:07:46 notifyme[8932]:      588 20011002220618.470 IDS|DDPLUS
631  SXUS70 KWAL 022203
Oct 02 22:07:46 notifyme[8932]:      640 20011002220618.817 IDS|DDPLUS
647  SXUS70 KWAL 022203
Oct 02 22:07:46 notifyme[8932]:      307 20011002220618.820 IDS|DDPLUS
648  SXUS70 KWAL 022203
Oct 02 22:07:46 notifyme[8932]:       99 20011002220618.823 IDS|DDPLUS
649  SPCA42 KWBC 022204
Oct 02 22:07:47 notifyme[8932]:       87 20011002220618.825 IDS|DDPLUS
650  SPUS41 KWBC 022204
Oct 02 22:07:47 notifyme[8932]:      147 20011002220618.834 IDS|DDPLUS
652  SPUS43 KWBC 022204
Oct 02 22:07:47 notifyme[8932]:      100 20011002220618.837 IDS|DDPLUS
653  SPUS54 KWBC 022204
Oct 02 22:07:47 notifyme[8932]:       99 20011002220618.840 IDS|DDPLUS
654  SPCA32 KWBC 022204
Oct 02 22:07:48 notifyme[8932]:      552 20011002220618.843 IDS|DDPLUS
655  SXPS82 RJTD 022000
Oct 02 22:07:48 notifyme[8932]:       98 20011002220618.868 IDS|DDPLUS
657  SNPS40 AMMC 022000 RRK
Oct 02 22:07:48 notifyme[8932]:      266 20011002220618.870 IDS|DDPLUS
658  SWPA41 RJTD 022200
Oct 02 22:07:49 notifyme[8932]:      658 20011002220618.873 IDS|DDPLUS
659  SAUK51 EGRR 022150
Oct 02 22:07:49 notifyme[8932]:      103 20011002220619.204 IDS|DDPLUS
671  SAPH00 RPLL 022200
Oct 02 22:07:49 notifyme[8932]:      624 20011002220619.208 IDS|DDPLUS
672  SAFR31 LFPW 022200
Oct 02 22:07:50 notifyme[8932]:      677 20011002220619.212 IDS|DDPLUS
673  SXUS70 KWAL 022203
Oct 02 22:07:50 notifyme[8932]:      387 20011002220619.237 IDS|DDPLUS
674  SXUS70 KWAL 022204
Oct 02 22:07:50 notifyme[8932]:    13690 20011002220749.012 NNEXRAD 792 
SDUS54 KMOB 022205 /pN0RMOB
Oct 02 22:07:50 notifyme[8932]:     3692 20011002220619.238 IDS|DDPLUS
676  FPUS65 KABQ 022206 /pSFPNM
Oct 02 22:07:51 notifyme[8932]:       91 20011002220619.239 IDS|DDPLUS
678  NZUS07 KPQR 022205 /pMTRFCG
Oct 02 22:07:51 notifyme[8932]:      107 20011002220619.239 IDS|DDPLUS
679  NZUS07 KPQR 022206 /pMTRFCG
Oct 02 22:07:51 notifyme[8932]:      110 20011002220619.240 IDS|DDPLUS
680  NZUS07 KPQR 022206 /pMTR92S
Oct 02 22:07:51 notifyme[8932]:      107 20011002220619.298 IDS|DDPLUS
688  NZUS07 KPQR 022206 /pMTRFCG
Oct 02 22:07:52 notifyme[8932]:      889 20011002220623.538 IDS|DDPLUS
702  SXUS70 KWAL 022204
Oct 02 22:07:52 notifyme[8932]:      883 20011002220623.545 IDS|DDPLUS
703  SXUS70 KWAL 022204
Oct 02 22:07:52 notifyme[8932]:      264 20011002220623.618 IDS|DDPLUS
711  NTUS99 KNCF 022206 /pTSTNCF
Oct 02 22:07:52 notifyme[8932]:      264 20011002220623.630 IDS|DDPLUS
713  NTUS96 KNCF 022206 /pWTSNCF
Oct 02 22:07:53 notifyme[8932]:     1721 20011002220623.716 IDS|DDPLUS
718  ASUS43 KBIS 022206 /pSWRND
Oct 02 22:07:53 notifyme[8932]:    17719 20011002220751.283 NNEXRAD 796 
SDUS54 KMOB 022206 /pN0REVX
Oct 02 22:07:53 notifyme[8932]:     6439 20011002220623.911 IDS|DDPLUS
723  ABAK32 PANC 022202
Oct 02 22:07:53 notifyme[8932]:      918 20011002220623.916 IDS|DDPLUS
724  SAUS70 KWBC 022200 RRC
Oct 02 22:07:54 notifyme[8932]:      664 20011002220623.920 IDS|DDPLUS
725  SAUS70 KWBC 022205
Oct 02 22:07:54 notifyme[8932]:      214 20011002220623.922 IDS|DDPLUS
726  SAUS80 KWBC 022200 CCD
Oct 02 22:07:54 notifyme[8932]:       93 20011002220624.443 IDS|DDPLUS
753  NZUS07 KPQR 022206 /pMTR4S1
Oct 02 22:07:54 notifyme[8932]:       93 20011002220624.445 IDS|DDPLUS
754  NZUS07 KPQR 022206 /pMTR4S1
Oct 02 22:07:55 notifyme[8932]:       93 20011002220624.447 IDS|DDPLUS
755  NZUS07 KPQR 022206 /pMTR4S1
Oct 02 22:07:55 notifyme[8932]:       93 20011002220628.569 IDS|DDPLUS
765  NZUS07 KPQR 022206 /pMTR4S1
Oct 02 22:07:59 notifyme[8932]:     1664 20011002220759.587 NNEXRAD 810 
SDUS84 KBMX 022159 /pDPABMX
Oct 02 22:07:59 notifyme[8932]:    17628 20011002220758.908     WSI
21791  NEX/YUX/VEL2/200110022200
Oct 02 22:08:00 notifyme[8932]:     2479 20011002220759.587 NNEXRAD 811 
SDUS54 KBMX 022159 /pNTPBMX
Oct 02 22:08:01 notifyme[8932]:     1210 20011002220635.261 IDS|DDPLUS
838  AWUS43 KLSX 022206 /pAWSSTL
Oct 02 22:08:01 notifyme[8932]:    18670 20011002220759.848     WSI
21792  NEX/YUX/SRMV2/200110022200
Oct 02 22:08:02 notifyme[8932]:    14330 20011002220801.375     WSI
21793  NEX/YUX/VEL3/200110022200
Oct 02 22:08:02 notifyme[8932]:     9786 20011002220802.154     WSI
21794  NEX/YUX/BREF3/200110022200
Oct 02 22:08:03 notifyme[8932]:       79 20011002220636.395 IDS|DDPLUS
853  SRUS75 KOTX 022206 CCA /pRR7LWS
Oct 02 22:08:04 notifyme[8932]:     3750 20011002220636.513 IDS|DDPLUS
861  ASUS45 KGTF 022206 /pSWRMT
Oct 02 22:08:04 notifyme[8932]:       79 20011002220645.393 IDS|DDPLUS
878  SRUS74 KOUN 022206 CCA /pRR7OKC
Oct 02 22:08:04 notifyme[8932]: Interrupt
Oct 02 22:08:04 notifyme[8932]: exiting

From the time differences I can see that FSU is only about two minutes
off with the IDS|DDPLUS feed.  

And, I'm *still* waiting for more to come from the 'notifyme' that I did
from your machine.

This indicates network trouble between your site and FSU.  Now, I'll
kill the notifyme and instead do a traceroute to FSU to see the times
that it reports:

[ldm@thermal ~/etc]$ /usr/sbin/traceroute  pluto.met.fsu.edu
traceroute to pluto.met.fsu.edu (128.186.5.177), 30 hops max, 38 byte
packets
 1  node129-254.unnamed.db.erau.edu (155.31.129.254)  0.533 ms  0.524
ms  0.429 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * *


This traceroute never completed - I killed it because it took way too
long.  These results are not good.  Asterisks indicate either a time out
or a firewall.  In your case, you can't even get a decent time getting
off your own campus.  Here's a traceroute to FSU from my own machine for
comparison:

  (anne) imogene:/home/anne/conferences/AMS 77 % traceroute
pluto.met.fsu.edu
traceroute to pluto.met.fsu.edu (128.186.5.177), 30 hops max, 38 byte
packets
 1  flra-n140 (128.117.140.252)  1.521 ms  0.282 ms  0.263 ms
 2  vbnsr-n2.ucar.edu (128.117.2.252)  0.833 ms  1.852 ms  0.954 ms
 3  internetr-n243-104.ucar.edu (128.117.243.106)  0.857 ms  0.828 ms 
0.711 ms
 4  denv-abilene.ucar.edu (128.117.243.126)  2.722 ms  2.351 ms  1.742
ms
 5  kscy-dnvr.abilene.ucaid.edu (198.32.8.14)  14.547 ms  12.887 ms 
13.047 ms
 6  ipls-kscy.abilene.ucaid.edu (198.32.8.6)  22.910 ms  21.653 ms 
21.879 ms
 7  atla-ipls.abilene.ucaid.edu (198.32.8.42)  32.493 ms  33.112 ms *
 8  a40010.c7507.fsu.edu (192.80.53.42)  38.653 ms  38.530 ms  41.708 ms
 9  vlan916.msfc.bfs.fsu.edu (128.186.250.17)  41.353 ms  38.100 ms 
40.178 ms
10  vlan964.msfc.dsl.fsu.edu (128.186.250.65)  250.739 ms  40.932 ms 
231.415 ms
11  pluto.met.fsu.edu (128.186.5.177)  116.604 ms *  125.867 ms

The times greater than 80ms are not that good.  But, regardless, we saw
that FSU was getting products in a relatively timely manner. 

Now, I'll do a traceroute to your machine from my own.  This would
reflect the path that a product would take from our server, motherlode,
to your machine.  (Maybe most of your products come from some other top
tier machine, but this is still a significant piece of data):

(anne) imogene:/home/anne/conferences/AMS 79 % traceroute
thermal.db.erau.edu
traceroute to thermal.db.erau.edu (155.31.129.19), 30 hops max, 38 byte
packets
 1  flra-n140 (128.117.140.252)  1.568 ms  0.289 ms  0.279 ms
 2  vbnsr-n2.ucar.edu (128.117.2.252)  2.564 ms  0.816 ms  1.050 ms
 3  internetr-n243-104.ucar.edu (128.117.243.106)  1.101 ms  0.727 ms 
0.749 ms
 4  frgp-gw-1.ucar.edu (128.117.243.114)  1.862 ms  1.613 ms  1.920 ms
 5  12.124.158.13 (12.124.158.13)  2.298 ms  2.012 ms  2.643 ms
 6  gbr1-p60.dvmco.ip.att.net (12.123.36.138)  2.143 ms  2.038 ms  1.977
ms
 7  gbr3-p70.dvmco.ip.att.net (12.122.5.17)  3.123 ms  2.489 ms  1.984
ms
 8  gbr4-p80.sffca.ip.att.net (12.122.2.65)  25.740 ms  25.674 ms 
26.033 ms
 9  ggr1-p370.sffca.ip.att.net (12.123.13.69)  26.128 ms  26.207 ms 
27.639 ms
10  svl-brdr-01.inet.qwest.net (205.171.4.233)  27.594 ms  28.875 ms 
27.473 ms
11  svl-core-03.inet.qwest.net (205.171.14.109)  27.822 ms  28.191 ms 
27.935 ms
12  kcm-core-01.inet.qwest.net (205.171.8.138)  49.147 ms  49.083 ms 
49.108 ms
13  kcm-core-02.inet.qwest.net (205.171.29.126)  49.246 ms  49.414 ms 
49.615 ms
14  atl-core-03.inet.qwest.net (205.171.29.154)  67.181 ms  67.885 ms 
67.319 ms
15  atl-core-02.inet.qwest.net (205.171.21.157)  67.382 ms  67.553 ms 
67.458 ms
16  atl-edge-14.inet.qwest.net (205.171.21.182)  67.144 ms  67.522 ms 
66.928 ms
17  atl-edge-14.inet.qwest.net (205.171.21.182)  67.184 ms  67.145 ms 
67.067 ms
18  208.46.254.134 (208.46.254.134)  749.457 ms  687.989 ms
208.46.254.206 (208.46.254.206)  118.481 ms
19  * * *
20  * * *
21  * * *
22  * * *
23  * *

I killed this one too.  These last times are B-A-D BAD.  My next query
is to identify the site where things go to pot.  (I had to do this from
a different machine that had the 'awhois' command that I wanted to run):

        (anne) laraine:/export/dist/ftp/pub/idd/ldmstats 53 % awhois
208.46.254.134
        Qwest Communications (NETBLK-NET-QWEST-BLK) NET-QWEST-BLK
                                                   208.44.0.0 -
208.47.255.255
        Qwest Communications (NETBLK-QWEST-TPA-SERIALS) QWEST-TPA-SERIALS
                                                 208.46.254.0 -
208.46.254.255

        To single out one record, look it up with "!xxx", where xxx is the
        handle, shown in parenthesis following the name, which comes first.

        The ARIN Registration Services Host contains ONLY Internet
        Network Information: Networks, ASN's, and related POC's.
        Please use the whois server at rs.internic.net for DOMAIN related
        Information and whois.nic.mil for NIPRNET Information.

AHA!  It's a Qwest site!

I'm not sure what this means.  It could be due to high traffic.  It
could be a hardware problem.  But, this connection problem is a
significant one.  Before I spend the time to look at other pieces of
your configuration, it's important to address this issue.  Were you
having connectivity problems before?

First, do reset your clock.  If you can get an NTP daemon running, that
would be best.  If you want info on how to do that, let me know.  But,
in the meantime you can simply reset the clock via 'date' or some other
simple tool.

Next, you must talk with your network engineers and describe the problem
to them.  Take them some traceroutes to show them the problem.   Maybe
you can get some info about the path (or paths) to FSU.  If you'd like
more data points from me, let me know.

There is one other thing you can do.  In your '~ldm/bin/ldmadmin'
script, you can change the default setting so that your LDM won't reject
products that are just over an hour late.  For example, you might want
to accept products that are two hours old.  I can make that change for
you if you want - I'll mark it in the file so you can see what I did.

Let me know how you would like to proceed from here.

Anne
-- 
***************************************************
Anne Wilson                     UCAR Unidata Program            
address@hidden                 P.O. Box 3000
                                  Boulder, CO  80307
----------------------------------------------------
Unidata WWW server       http://www.unidata.ucar.edu/
****************************************************