[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #NXJ-554265]: LDM setup questions

Hi Yoori,

re:
> I think I completed parts what you mentioned that I was missing. I went 
> through whole LDM
> documentation again to make sure I didn't miss any steps.

Very good.

> Yes, I created crontab and when I check by typing 'crontab -l', I could see 
> what I added.
> 0 * * * * bin/ldmadmin scour >/dev/null
> 0 0 * * * bin/ldmadmin newlog

I am glad that you are able to edit your crontab entries now.  I have come 
comments,
however:

1) your crontab entry will run 'bin/ldmadmin scour' every hour.  This is 
probably
   excessive, so I would suggest doing it once every three hours.

2) your first crontab entry pipes the output from STDOUT to /dev/null, but it 
does
   not do anything with STDERR.  Unidata sites typically pipe both output to
   /dev/null

3) your second entry does not do anything with possible output.  Users typically
   pipe the output to /dev/null

4) your second entry will be run at just after midnight local time (unless your
   computer's clock is running on UTC which is unusual; AND your machine is not
   running on UTC).

Comments:

- Piping the output of actions run from cron to /dev/null will keep your system 
from
  mailing 'ldm' with the output of each execution.  This may not seem important 
now,
  but you will see that this generates a LOT of mail.

- There is another way that you can turn off the mailing of status messages from
  cron actions, inclusion of MAILTO="" at the top of crontab.

- I suggest running the second cron file after 0 UTC, not 0 local time

Given these two things, I suggest the following content for your crontab file:

MAILTO=""
0 0,3,6,9,12,15,18,21 * * * bin/ldmadmin scour
0 20 * * * bin/ldmadmin newlog

This can be written more compactly as:

MAILTO=""
0 0-23/3 * * * bin/ldmadmin scour
0 20 * * * bin/ldmadmin newlog

Please review the system man pages on crontab for more information on what can 
be
done in your crontab file.

> And I added the below lines at ~ldm/etc/ldmd.conf
> REQUEST       IDS|DDPLUS      .*      workshop19.ucar.edu
> REQUEST       NNEXRAD         .*      workshop19.ucar.edu

A quick comment here:  The machine 'workshop19.ucar.edu' is listed in LDM 
documentation
as an example entry.  It is not a real machine, so your request for data from 
it will
fail.  Because of this, I changed your two entries to request data from the 
Unidata-maintained
toplevel IDD relay node at the NSF offices in Virginia.  The following 
represents the
current state of request lines in your ~ldm/etc/ldmd.conf file:

# IDS|DDPLUS - Gobal observational data
REQUEST IDS|DDPLUS      .*      idd.cise-nsf.gov

# NNEXRAD    - NEXRAD Level III products
REQUEST NNEXRAD         .*      idd.cise-nsf.gov

# Added EXP upstream feeders
REQUEST EXP "^NEXRAD4" mapserver.unidata.ucar.edu

> From your sentences, "The files will not be processed unless you have one or 
> more
> actions in your ~ldm/etc/pqact.conf file.  I noted last night that I did not 
> see
> an action for the data you were requesting from mapserver.unidata.ucar.edu, 
> so nothing
> is being done with the data you are receiving (it simply is sent from the
> upstream machine (mapserver.unidata.ucar.edu) and is received into the LDM 
> queue on
> your machine)."

Your ~ldm/etc/pqact.conf file _still_ does not have any entries that would 
process
the data from the EXP feed you are requesting from mapserver.unidata.ucar.edu.

> => I think I don't have any more steps left.

You do.  You have to figure out what it is you want to do with the EXP data you
are requesting from mapserver.unidata.ucar.edu.

> After I completed all missing parts, and then type the command for 'ldmadmin 
> restart'
> and then 'ldmadmin watch' to see whether the data is ingesting. It seems 
> working, but
> still in the none of place (directory) in my computer, I cannot see the data 
> from 
> mapserver.unidata.ucar.edu.

Correct, except you had two problems:

1) you were requesting IDS|DDPLUS data from a non-existent machine.

2) your ~ldm/etc/pqact.conf file has an action to process IDS|DDPLUS
   data only

3) your ~ldm/etc/pqact.conf file has _no_ action to process any of
   the EXP data from mapserver.unidata.ucar.edu

> I thought if ldm is working correctly to ingest from upstream 
> (mapserver.unidata.ucar.edu)
> to mysite,

Data is being received on your machine; 'ldmadmin watch' shows you this is true.

> the data should be stored in somewhere in my computer, but it was Not in case.

The data will be processed out of the LDM queue as soon as you put an action in
~ldm/etc/pqact.conf that does the processing.

What may not be apparent is the LDM is composed of two parts:

1) Data receipt and relay
  
  - request data from one or more upstream machines
  - provide data to zero or more downstream machines that you have ALLOWed
    to request data from you

2) processing of data received

  - this is accomplished through entries in ~ldm/etc/pqact.conf

What the actions in ~ldm/etc/pqact.conf are depend completely on what
you want to have done.

> I'm kind of afraid that I still miss some steps.

I think what you are missing is the "big picture overview" of what the LDM
is and how the various pieces fit together and are configured.

> Sorry for the continuous troubles and thanks for your help.

No worries.  Installing and configuring an LDM without the benefit of attending
a training workshop is difficult.  Along these lines, we will be holding
a training workshop series this coming July/August.  You can learn more about 
the
training workshops on our homepage:

http://www.unidata.ucar.edu

I suggest you talk to your department about sending you to the LDM training
workshop.

By the way, I will be setting up automatic starting of the LDM in the next hour
or so.  After I make the needed additions to /etc/init.d, I will want to reboot
your machine to test the changes.  Is there any problem in doing this?



Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: NXJ-554265
Department: Support LDM
Priority: Normal
Status: Closed