[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20010629: ewbie question

Bunny Pfau wrote:
> 
> -> Date: Mon, 02 Jul 2001 13:23:21 -0600
> -> From: Anne Wilson <address@hidden>
> -> X-Accept-Language: en
> -> MIME-Version: 1.0
> -> To: Bunny Pfau <address@hidden>
> -> CC: address@hidden
> -> Subject: Re: 20010629: ewbie question
> -> Content-Transfer-Encoding: 7bit
> ->
> -> Unidata Support wrote:
> -> >
> -> > ------- Forwarded Message
> -> >
> -> > >To: address@hidden
> -> > >From: Bunny Pfau <address@hidden>
> -> > >Subject: ewbie question
> -> > >Organization: UCAR/Unidata
> -> > >Keywords: 200106292145.f5TLjw107630
> -> >
> -> > Hello:
> -> >
> -> > I'm using LDM 5.0.8 on Solaris 7.  The question I have
> -> > I could not find answered in the documention, although
> -> > I bet the answer IS there.
> -> >
> -> > My question is, when a data file is arriving on a client
> -> > machine, it appears that the datafile arrives ALL AT ONE
> -> > TIME.  See, I'm afraid that a program that I have that
> -> > attempts to read the newly arrived file *might* try
> -> > to access thefile before it has completely arrived--
> -> > like it's only HALF there.
> -> >
> -> > Tests I've done seem to indicate that some sort of
> -> > caching happens, thus the file that lands in my
> -> > data directory, materializes all at once!  I hope
> -> > I'm getting my message across here--it's kind of
> -> > hard to put into work^R
> -> > hard to put into words..
> -> >
> -> > Anyway, does the way that LDM works, assure me
> -> > that the requested data file arrives all at once?
> -> >
> -> > Thanks,
> -> > Bunny Pfau
> -> > HAO
> -> >
> -> > ------- End of Forwarded Message
> -> Hi Bunny,
> ->
> -> In the context of the LDM there are two possible "arrivals" for a
> -> product.  First is when a product arrives and is placed into the product
> -> queue.  Second is when pqact processes a product in the queue and writes
> -> it to a file or to a decoder.  I assume you're talking about when pqact
> -> processes a product in the queue and writes it to a file.
> ->
> -> The answer to your question is: no.  You can't be sure that simply
> -> because the file exists it is all there.  Figuring out when a file is
> -> complete can be hard.  Programs invoked by pqact (part of the LDM) that
> -> process the data usually (a) know something about the products so that
> -> they can identify the end of the product, and/or (b) use timing in some
> -> way to decide whether there will be more data coming or not.  If you're
> -> interested let me know and I'll show you a few examples.
> ->
> -> Anne
> -> --
> -> ***************************************************
> -> Anne Wilson                  UCAR Unidata Program
> -> address@hidden                       P.O. Box 3000
> ->                                        Boulder, CO  80307
> -> ----------------------------------------------------
> -> Unidata WWW server       http://www.unidata.ucar.edu/
> -> ****************************************************
> 
> Anne:
> 
> Thanks for your answer.  I appreciate it!
> 
> I would indeed be interested in the examples!
> 
> Thanks,
> Bunny Pfau
> HAO

Hi Bunny,

The best examples are the decoders that we make available in our decoder
package.  You can get that at
http://www.unidata.ucar.edu/packages/decoders/index.html.  In this
group, gribtonc is written in C.  I think the rest are written in Perl. 
These can serve as templates for you.

Above I implied that decoders may use timing *only* to determine the end
of a product.  This is not the case.  Decoders must either be able to
recognize product boundaries, or products must be sent to them with a
'PIPE -close' so that an end of file (EOF) character is tacked on to the
end of the product, which a decoder can recognize.  For example, netcdf
files have no terminating byte string, so they must be sent to decoders
with a 'PIPE -close'.  On the other hand, GRIB products start with
'GRIB' and end with '7777' and also provide intermediate block sizes so
that '7777' may appear inside the product.  

(Timing is used in decoders that can recognize product boundaries, as
opposed to keying off an EOF character, to keep them alive as long as
products are arriving.  This way one decoder can process multiple
products, and it will only kill itself when the input stream has been
idle for some time.  Our decoders will generally die after the stream
has been empty for 20 minutes.)

I hope this helps.

Anne
-- 
***************************************************
Anne Wilson                     UCAR Unidata Program            
address@hidden                 P.O. Box 3000
                                  Boulder, CO  80307
----------------------------------------------------
Unidata WWW server       http://www.unidata.ucar.edu/
****************************************************