[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [awipsldm] LDM Observations and Comments

Russ,

Ken must think UPC really likes him.  We both answered him today on the
same question. Strange, he sent the message to me directly and I didn't
get the message via awipsldm list even though I'm on the list. The good
part is that we both dwelt on different aspects of the question. I'm 
working on the LDM statistics today so my shorter answer pointed to
creating LDM compatiable perl scripts instead of RE.  

Robb...

ps It was correct to send the response to the whole list awipsldm




On Tue, 25 Jan 2000, Russ Rew wrote:

> Hi Ken,
> 
> I don't know whether this reply is appropriate for the whole AWIPS LDM
> list, but feel free to forward it there if you think that would be
> useful.
> 
> > For one, I was looking for the capability to do use a Boolean "NOT" in my 
> > pattern matching.  For instance, send all products EXCEPT for MTRs and 
> > TAFs.  As far as I can tell, that capability does not exist.  The "OR" 
> > operator is there (a pipe '|') but not a "NOT" operator.
> 
> This limitation comes from the lack of a "NOT" operator in regular
> expressions.  It's always possible to write a regular expression that
> represents the complement of any specified regular expression, but the
> complement regular expression can be much longer than the original.
> For example, a regular expression matching either of the strings "MTR"
> or "TAF" is just:
> 
>   MTR|TAF
> 
> but a regular expression that represents all three-character strings
> other than "MTR" or "TAF" is considerably longer and not obvious:
> 
>   [^MT]..|M[^T].|MT[^R]|T[^A].|TA[^F]
> 
> So although a "NOT" operator is not theoretically necessary, it is a
> practical necessity.  (That's why the "-v" flag was added to the
> "grep" utility in Unix to "Print all lines except those that contain
> the pattern", for example.)
> 
> We tried to add something similar to the LDM with the "_ELSE_"
> pattern, which provides a place to invoke an action on products
> that so far have not matched any pattern, as in
> 
>   WMO  MTR    action for MTR products
>   WMO  TAF    action for TAF products
>   WMO _ELSE_  action for products that have matched nothing else so far
> 
> but this really doesn't work very well if you need more than one
> "_ELSE_" pattern.
> 
> I think adding a NOT operator is a good idea, but am not sure what
> notation to use; perl5 added "(?!...)" for this purpose where "..." is
> a regular expression, but perl regular expressions are probably too
> complicated for LDM use.  Another alternative would be to use some
> character such as "!" at the beginning of the action section to mean
> do this only if the product didn't match the pattern (but did match
> the feedtype), so it would appear something like:
> 
>  WMO  MTR|TAF    !FILE other-prods
> 
> to file all WMO products that don't match MTR or TAF in the
> other-prods file, for example.  This would not be too hard to
> implement, if users think it would be useful and if we can find a way
> to do it that doesn't conflict with current LDM notation.
> 
> > Another observation is that the append function (i.e., not using the
> > -overwrite option) always appends to the bottom of the file, using
> > standard Unix appending.  I would like to append to the top of the
> > file.  There is no way to do this other than writing scripts and
> > appending manually.
> 
> There is no way to do this efficiently with Unix I/O calls; it
> requires opening a new file, writing the new product to the new file,
> copying the old file to the end of the new file and then renaming the
> new file to the old file name, a potentially expensive operation if
> the original file is large.  Using scripts is about as efficient as
> making this a built-in LDM operation.
> 
> > Okay, what am I getting to on all this?  Well, my concern is how the
> > operating system can handle so many calls to scripts and I'm
> > concerned that some will get lost due to multiple calls to the same
> > script with possibly the same matching pattern.  Is it better to
> > just use LDM to write out the product to a temporary file and then
> > set up a PIPE to do all the exception handling above?
> 
> Each script starts up a process, but these days computers can start
> many processes per second, and you typically only get 5 or 10 products
> per second, so I don't think this will overload any modern platform
> that's not already overloaded with lots of other work.  If something
> got lost, that would be a bug in pqact, and we'd like to know about it
> so we could fix it.
> 
> It's better not to write a temporary file if you don't have to, since
> creating a new file, especially in a directory that already has lots
> of files, can be as expensive as starting up a process.
> 
> Using the PIPE action with decoders typically doesn't have to write a
> temporary file or start a new process, as long as the decoder follows
> the convention of reading its input from stdin, and sleeping until
> there is more data on stdin rather than terminating.  pqact keeps a
> list of active decoders and routes products to existing decoders when
> possible.  If a decoder hasn't been used for a long time and a new
> decoder is needed, pqact closes the pipe to that decoder, which tells
> the decoder to terminate gracefully.
> 
> You may be able to write your scripts like decoders to follow these
> same conventions; that's what Robb Kambic did with the perl decoders,
> which you can look at for examples.
> 
> Anyway, thanks for the good suggestions; we'll definitely consider
> adding the "NOT" operator to the next version if it's practical.
> 
> --Russ
> 
> _____________________________________________________________________
> 
> Russ Rew                                         UCAR Unidata Program
> address@hidden                     http://www.unidata.ucar.edu
> 

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================