[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20000630: LDM setup at INM (cont.)

>From: Pepo Juega <address@hidden>
>Organization: Instituto Nacional de Meteorologia
>Keywords: 200006220829.e5M8TZT11974 LDM ldm-mcidas

Pepo,

re: many machines being fed off a single server

>I don't have that clear yet. In order to have a series of machines
>that receive from say meteosat, I have to:
>-include proper downstream hosts entries in netcheck.conf
>-have corresponding ALLOW entries in ldmd.conf
>-don't want any pqact started here (?)

You basically only have to expand on what is already working.  From
previous emails, the ~ldm/etc/ldmd.conf setup that I think you are
using on the "server", meteosat.inm.es, is:

exec    "pqexpire -i 300"

allow   ANY
        ^((localhost|loopback)|(127\.0\.0\.1\.?$))

allow   ANY     alfa.inm.es


This simply needs to be expanded to 'allow' each machine that you are
willing to feed.  Let's assume that the list of machines is:
hail.inm.es, tornado.inm.es, cyclone.inm.es, nubles.inm.es.  'meteosat's
ldmd.conf will need to be modified to include these machines by
allows:

exec    "pqexpire -i 300"

allow   ANY
        ^((localhost|loopback)|(127\.0\.0\.1\.?$))

allow   ANY     alfa.inm.es
allow   ANY     hail.inm.es
allow   ANY     tornado.inm.es
allow   ANY     cyclone.inm.es
allow   ANY     nubles.inm.es

As you really start using the LDM, you may want to refine these entries.
For instance, assume that you decide to start shipping more than just
McIDAS AREA and GRID file data around.  Suppose you have some locally
generated experimental model data that you want mainly sent to a different
list of machines.  Your ldmd.conf file might end up looking like:


exec    "pqexpire -i 300"

allow   ANY
        ^((localhost|loopback)|(127\.0\.0\.1\.?$))

allow   MCIDAS  alfa.inm.es
allow   MCIDAS  hail.inm.es
allow   MCIDAS  tornado.inm.es
allow   MCIDAS  cyclone.inm.es

allow   MCIDAS|EXP      nubles.inm.es
allow   MCIDAS|EXP      mammatus.inm.es

allow   EXP     storm.inm.es
allow   EXP     front.inm.es
allow   EXP     cirrus.inm.es


After stopping and restarting the LDM on meteosat (ldmd.conf is only
read at LDM startup), you will be setup to feed the McIDAS AREA and
GRID data (actually, any data tagged as being in the MCIDAS feed; the
type of data does not matter at all) to one set of machines and feed
the experimental model data to a different set of machines.  Note how
you can say that you are willing to feed either MCIDAS or EXP to nubles
and mammatus.

Perhaps you want to allow any machine in the inm.es domain to be able
to request data in the MCIDAS feed.  Your ldmd.conf entries could be
greatly simplified as follows:

exec    "pqexpire -i 300"

allow   ANY
        ^((localhost|loopback)|(127\.0\.0\.1\.?$))

allow   MCIDAS  ^[a-z].*\.imn\.es$

allow   EXP     nubles.inm.es
allow   EXP     mammatus.inm.es
allow   EXP     storm.inm.es
allow   EXP     front.inm.es
allow   EXP     cirrus.inm.es

Just to make sure that you are not misled by the names of the feeds
(e.g., MCIDAS, EXP), they really are just a way of naming a collection
of data sent under a single stream.  I think you can see the flexibility
offered by being able to describe things with regular expressions.

Your question about netcheck.conf was a good one.  netcheck is used
to monitor network connectivity to select hosts.  Its configuration
has nothing to do with who you set yourself up to feed.

For my own etification, I provide:

exec    ->  programs to run at LDM startup

allow   ->  allow a host or hosts to request data from me.  Without
            allowing a host, a request from it will be denied

            The local host should _always_ be allowed!

request ->  ask a server to feed me products of a certain type (i.e.,
            products grouped by a particular feed type) that match
            a regular expression that is provided (so that a "client"
            may request a subset of the products in a particular
            feed (useful for slow machines; machines with not much
            disk space; or machines that have a poor network
            connection)

An LDM machine's ldmd.conf can have multiple allows as shown above,
and it can also have multiple requests.  This feature allows a machine
to act as a data relay.

>At the receiving machine, say alfa:
>-include meteosat as my upstream host in netcheck.conf
>-request entry for MCIDAS feedset from meteosat.
>-a nice pqact.conf and pqact running

Again, the netcheck.conf part is useful but not necessary.  In
alfa's ldmd.conf you 'request' data from a particular data stream
(e.g., MCIDAS, EXP, etc.) whose headers match the regular expression
in the request:

request ".*"    meteosat.inm.es

The '".*"' is a regular expression that matches everything.

On the receiving machine you will definitely need to have pqact running.
Otherwise, the data simply is received in the queue and sits there.

>Is that all? As for the product queue at meteosat. I thought that
>what scour does is delete old files from the data subdir, and
>now I wonder what happens with products inside the QUEUE (ldm.pq)

You are correct.  'scour' does scour data files in disk directories.
To get products out of a queue, you run pqexpire.

>They are kept in the queue even though transmission was successful?

Yes, until they are 'expired' out of the queue by pqexpire or until
the queue is so full that one or more products has to be deleted
to make room for new ones.

>What if some of the recipients did not get it? How do I find out?

The ldmd.log file will tell you if there are problems sending products
to a recepient.

>How does retention/deletion at the queue work?

The product stays in the queue until it is 'pqexpire'd.

>(I got some
>messages saying pqinsert: Product already in queue)

Right.  If you try to insert the exact same product into a queue
that already has it, the insert is rejected since it is not needed.

>What attributes count to consider a product already in the queue?
>(file name, feedtype, date, age...?

The product header and the MD5 checksum that is created from the
contents of the product.

>yes but what if I use the same
>file name with different contents for successive transmissions?)

No problem.  The MD5 checksum will be different.

>I looked at pqexpire's man page, but it only explains the essentials

If you are only putting a few products into the queue, then you
might consider running pqexpire more frequently.  What you need
to judge is how slow your slowest link is.  You will want to make
sure that the product(s) that are to be sent over the slow link
are not expired before they are delivered.  You control how old
a product in the queue has to be to get expired by specifying
the '-a' flag to pqexpire.

>On the other hand I fed the queue using pqinsert, but I read about
>pqing and pqsend...

Yes.  pqsend can be used to send a product in a local product queue
directly to a single LDM server.  -- aside comment --  the use of
client and server in the LDM context may, at first glance, seem
to be reversed.  The way to think about this is the following:

o the machine making the request for data is the server: it registers
  a request for transmission of data from a client as soon as the
  client gets data that looks like that requested (e.g., data feed
  type and product header regular expression match)

o the client is actually the machine that sends the data: the reason
  that it is referred to as a client is that it is passive; it will
  not send data unless there has been a request made to send the
  data

>I have to read the manual in depth, so don't
>worry too much about all this. Let me play a few days with all that 
>I got now, and we'll recap then.

I think that as you play with this, you will quickly get the hang
of it.  At that point, you will undoubtedly get a load of ideas
related to setting up your own IDD among your meteorlogy centers.

>I also have to navigate through your inquiry tracking system, BTW
>congrats, It's great.

Thanks.  It is servicable right now and promises to get a lot better
with some of the work being done currently.

>>re: PNG compression/uncompression routines
>
>OK, I built up your ldm-mcidas-7.6.3 which only complained about a
>CALL FLIPRTE in nids2area.f

I should have warned you about that.  The implicit assumption is
that one will be linking against the Unidata McIDAS library, not
the SSEC one.

>I commented those out of the source
>and even got away with make test.

Good move.  You don't need nids2area anyway.

>But thought that you would like
>to know.

I always appreciate feedback; thanks.

>I also got the binary release, and just overwrote the
>executable for nids2area, which I even think I will not be needing.

Right, you won't need it.

>Now I have what I wanted, pnga2area and area2png. What about a
>png2grid and a grid2png? do they exist?

There is no png2grid or grid2png.  I didn't write one since we are
not sending McIDAS GRID files in our IDD anymore.  If you write
this pair of routines, I will consider including them in a future
ldm-mcidas release.

re: ADDE access to METEOSAT data

>Ummmh! That will take us some time. Ask Doris, she knows well.

Ah, so Doris is getting the data...

>The best I can do is promote your helpful support to our Admin
>staff in the hope that it will move their reckless hearts to
>allow you in.

Thanks, I appreciate that!

>(It took Doris one whole year to get it !)

I am patient :-)

>Don't you get that through the folks at SSEC?

No, not really.  We can get some, but not on a regular basis.  What
I am trying to do is establish mutual agreements for sharing data.
This is part of the community building objective of Unidata.

>Txs, have a great weekend, eat lots of pizza.

Thanks.  Talk to you soon...

Tom