[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20050805: LDM and the SCOOP project

>From:  Gerry Creager <address@hidden>
>Organization:  TAMU
>Keywords:  200508051331.j75DV7jo008831 LDM IDD SCOOP

Hi Gerry,

>I'm sitting in a SURA Coastal Ocean Observing and Prediction project 
>(SCOOP) meeting in Portland, ME, right now, and the topic's turned to 
>"problems with LDM".  We've seen some problems with delays, perceived 
>configuration problems, complaints about config files, dropped data, 
>supposed security weakness, etc.

Interesting.  We have talked about the need to carve up large model
output files into individual GRIB messages several times in phone
calls.  Has anyone done anything about this yet?  I think that this
step would likely eliminate concerns about:

- delays
- dropped data

As far as security weakenesses, all I can say is that to our knowledge
there has never been a security breach through the LDM.  Given this, I
would term the security weakness issue as "imagined" or "possible" not
"actual".

>I'd like to see if we could talk about this sometime soon, even if it 
>means me coming up there in Aug or late SEP to sit down and talk.

OK.  As you know now, Steve is currently on travel in Scotland.  He is
scheduled to return to the US on August 16.  I just checked his
calendar and see that he is occupied on August 18, so the window for
talks with him included opens on August 19.  I, on the other hand, will
be taking some vacation starting on Saturday, August 20.  I will return
to the office on Thursday, August 25 or Friday, August 26.

I am not sure of the scope of the discussions you want to have, but it
could be the case that the dialog could be accomplished by conference
call with Steve Chiswell, Mike Schmidt (our system administrator), and
myself.

>One of my reasons for promoting LDM for the data movement in SCOOP is 
>simple: A lot of supposed automation in Opendap still requires operator 
>intervention for data retrieval... or crontabs that have to be tweaked 
>more often than LDM configs.  The oceanographers have been doing 
>business for years in this manner, for the most part, and there's some 
>inertia associated with changing how they get their data.

OK.

>In other cases, with really big files, and this is something Tom and I 
>have talked about, checksumming and large file transmission is an issue. 

Yup, they should be carved up into smaller chunks and/or you should
create a different method of creating MD5 checksums for the products.

>  We may need some help getting things optimized over the next couple of 
>months refining what we're doing if we're really going to continue to 
>use LDM in future SCOOP activities.

OK.  We are more than willing to help you get the SCOOP data distribution
working smoothly.

>I'll admit that my predisposition to LDM for these data are predicated 
>on my somewhat greater familiarity with it than other technologies for 
>moving near real time data.
>
>What are the prospects for this?  I'll be in Boulder Mon evening-Thurs 
>afternoon for the MMI workshop, but I have *no* idea of evening 
>activities.  I'll know more Tues about 0815...

I will be in the office next week, but Steve Emmerson will not.

Cheers,

Tom
--
+-----------------------------------------------------------------------------+
* Tom Yoksas                                             UCAR Unidata Program *
* (303) 497-8642 (last resort)                                  P.O. Box 3000 *
* address@hidden                                   Boulder, CO 80307 *
* Unidata WWW Service                             http://www.unidata.ucar.edu/*
+-----------------------------------------------------------------------------+