[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

19990405: survey: LDM data recovery



>From: David Wojtowicz <address@hidden>
>Organization: University of Illinois
>Keywords: 199905050346.VAA11109 data archive

David and Harry,

I was going to write you anyway, so David's survey request was very timely.

Unidata is entering into a joint project with NCAR/SCD that is aimed at
making NCAR's massive data archive available to the Unidata community
free-of-charge.  The initial steps in this project are the installation
of the LDM on a new compute server (16 CPU SGI Origin 2000).  In the
account we are setting up, we will initially be allocated 8 GB of disk
space for "archiving" near realtime data from the IDD.  Archiving is in
quotes since the mechanism for getting the majority of data into the
NCAR Mass Store involves NCAR getting tapes from NCDC on a monthly
basis.  Their archive will not contain IDD data per se, but should
contain all of the data in the IDD with the possible exception of the
imagery from the Unidata-Wisconsin datastream.

The long term goal of the cooperative project is the production of a
system that allows Unidata users to request data like the NCEP
reanalysis from the NCAR Mass Store.  What form this will take is part
the work for the project.

The reason I was going to write to you was to get an idea of how you
have setup your systems to store the archived data that you make
available; how you allow users access to the data; and any comments on
the experience that you have had.  I see from David's survey that the
last item would probably be best answered after he hears from sites
using your archive.

Any/all input is most welcome.

>For several years here we've maintained an FTP site for IDD members
>to recover missed/lost IDD data from.
>
> In recent months though the greatly increased data volume from
>NOAAport and larger McIDAS and HDS data files combined with limited
>space on the machine hosting it has rendered it less than useful.
>
> I'm setting up a new archive site, but would like your input.
>Many have asked for and been given access, but I rarely hear if they
>find it useful or otherwise.
>
>Currently, the data is stored in raw feed files by feed by hour.
>The idea is that you download the hours and feeds that you need
>and then use the LDM ingestor programs to refeed the data into your
>local LDM product queue and have it processed by means of your
>local pqact actions just the way current data is.
>
>However, I'm not sure many people are aware of the means of doing this
>or if they've constructed their pqact.conf files to deal with anything
>other than the most recent data.  Nor is this method useful with
>the MCIDAS feed since there is no ingestor supplied for that feed
>supplied with LDM.
>
>
> 1) Do you use the archive site ftp://address@hidden/
>    (Note: there's not much there at the moment)
>
>
> 2) If so, was it at all useful?
>
> 3) Do you favor an approach that would let you refeed data into
>    your local ldm for local pqact processing?
>
>   -or-
>
> 4) Would you prefer just fetching data files in some more
>    readily useable form? such as GEMPAK?
>
>    GEMPAK works well because most Unidata sites use the same
>    GEMPAK file naming conventions as that's how Garp expects them.
>    It also seems to be a popularly requested format in "needdata".
>
>    Other formats?  WXP?  (but there's no one universal convention)
>
>Any other suggestions, comments, discussion are appreciated.
>
>Thanks.
>
>--------------------------------------------------------
> David Wojtowicz, Systems Manager
> Department of Atmospheric Sciences Computer Services
> University of Illinois at Urbana-Champaign
> email: address@hidden  phone: (217)333-8390
>--------------------------------------------------------

Tom
--
+-----------------------------------------------------------------------------+
* Tom Yoksas                                             UCAR Unidata Program *
* (303) 497-8642 (last resort)                                  P.O. Box 3000 *
* address@hidden                                   Boulder, CO 80307 *
* Unidata WWW Service                             http://www.unidata.ucar.edu/*
+-----------------------------------------------------------------------------+