[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Bouncing Ideas

Daryl Herzmann wrote:
> 
> Hi Anne,
>         I hope you do not mind the direct email, but I have some ideas to
> bounce off somebody.  And since nobody here at Iowa State seems to know
> much about LDM, I will pick your mind.
> 
>         I have been working on the Workstation ETA project for the Weather
> Service here in DSM. As a part of the project, I have been running the
> model on linux boxes here at Iowa State. So far so good, but I have
> bigger plans in mind.
> 
>         One of my ideas is to distribute the WSETA to many machine here on
> campus and at the weather service and have each machine run a varying
> version of the model (physics, resolution, dt, etc...). The problem is
> then passing all of the correct initialization data out to the workers and
> then collecting the results.  I looked at using NFS, SSH(scp) and rsync,
> but I thought they where too hard to configure and not robust enough. So
> then I started playing with LDM and have been very impressed so far.
> 
>         Currently, I have a worker machine pqinsert the gempak output file
> and send it back to my analysis server.  One of the questions I have is
> how to use a logical naming convention. For instance, this morning's 00z
> ETA run comes back from each worker as 00081700_wseta.gem, so I have an
> overwritting problem. Then I figured that I would just have each machine
> name their files different, so the server would store each on
> seperate. This was ok, but I would rather do something different. I quess
> my question is, when I use pqinsert the data comes accross as
> 
> "Aug 17 17:21:51 pqutil:  2663936 20000817171922.950     EXP 000
> 00081200_abe_wseta.gem"
> 
>         I would like to be able to do something like the text products
> come by as.
> "Aug 17 17:11:26 pqutil:      907 20000817171047.997 IDS|DDPLUS 230
> WUCN10 CWUL 171700"
> 
>         meaning my ideal file would be
> "Aug 17 17:21:51 pqutil:  2663936 20000817171922.950     EXP 000 00081200
> KFR abe
> 
>         So then I know that the file was created on the machine called abe
> and using the Kain-Fritcsh Cumulus scheme. Then I could use the pqact file
> to organize the returned data files a little more logically. Can this be
> done with pqinsert without naming the file differently?
> 
>         hope this all makes a little bit of sense.  Am I going about my
> project correctly? Am I utilizing LDM capabilities?  Any and all
> suggestions will be greatly appreciated.
> 
>         I am still a novice with LDM, so if a "READ THE MAN PAGES!" is
> required as an answer, please let me know :)
> 
> Thanks much,
>         Daryl
> 
>                                                     __   ,,,,
>                                                    /  \_|$$$$|__,
>                                                    |     ==========}
> ,,,,,, Daryl Herzmann  [ JR - MTEOR ]   ,,,,,,,,   +--+ JOHN    |_
> !!!!!! address@hidden              !!!!!!!! ___  |  DEERE ,_/
> $$$$$$ www.public.iastate.edu/~akrherz  $$$$$$$$/__|==-__-----_|
> |||||| ICQ: 25560887 AOL: "Daryl Herz"  ||||||||_______{}_____o_____

Hi Daryl,

My apologies for the slow response.  Somehow I
overlooked this one.

I think I understand the gist of your problem
(although maybe not the details).  However, the
bad news is that pqinsert will only insert the
product into the queue with the file name as the
product identifier.  If you want to use pqinsert
you will need to ensure the file name is in the
form you want it before you pqinsert it.  You
might be stuck with having each machine name their
files differently.

I'm going to speak with my coworker about your
question.  Perhaps there is an alternative, but I
would expect that it wouldn't be simple.  I will
let you know what I find out.

Otherwise, it seems like a fine use of the LDM. 
It would serve well as a way to gather all the
files from a variety of hosts.  pqact could
certainly be useful in organizing the files.  If I
understand your problem correctly, if you simply
appended the host name to each file (as it looked
like you were doing) then you could group all the
files with the same prefix together into a common
directory.

If I don't seem to be understanding this correctly
please let me know.  I'm happy to discuss this
further.

Anne


-- 
***************************************************
Anne Wilson                     UCAR Unidata Program            
address@hidden                 P.O. Box 3000
                                  Boulder, CO  80307
----------------------------------------------------
Unidata WWW server      
http://www.unidata.ucar.edu/
****************************************************