[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDD #TTR-495389]: Slow access to local datasets through ADDE

Hi Samuel,

re:
> O.K. Thank you for your help, it's been very insightful.

I hope that the process of setting up McIDAS ADDE datasets is not as
mysterious as it once was :-)

> I had an inkling thing about NEXRAD2, and already had setup a NFS share
> of /data.

Very good.

> I will work on making 'CURDAY' part of our slower datasets.

This should be a very useful construct for you for the NIDS, GINI, and NEXRCOMP
datasets.

> I suppose we can get along without CURDAY working for our 'AREA' format files.

I see that there currently 1307 images in the RTIMAGES/GE-IR dataset at the
moment on shu.  Datasets of this size always take a long time to list and
get data from, so working towards access to a subset through the daily directory
idea should be useful especially on a machine like shu given that it is
overloaded with data ingest processing at the moment.

> I simply chose ANTARCTIC, because it was one of our least used datasets.

I understood that.  The three sets of data in RTIMAGES that are very infrequent
are the Antarctic IR composites, and the Mollweide IR and WV composites.  Each
of these images is received every three hours.  The Vis, and various IR images
from GOES-East and GOES-West are sent every 15 minutes and are much larger so
the impact is much greater.

> I'll also work on using scourBYday in our cleanup procedures for the datasets
> that have been converted to use CURDAY.

Very good.

> Thank you for your help.

No worries.

> If it's o.k. I think I'll turn off the 'ALL' access to the subnets I mentioned
> earlier.  Nothing personal, I simply like increased security over weekends.

I would prefer that you leave SSH and ADDE access open to our domains so that I
could test the code mods I will be making to add the \CURDAY support to image
datasets of type AREA (do a lot of code development late on Saturday nights),
but I will understand if you choose to close access.  By the way, to my 
knowledge
there has never been a security breach through ADDE, and there is no way to
"discover" ADDE sites, so leaving port 112 access open to the world should cause
no problems or add any load to your machine.  I always request that my McIDAS
sites open ADDE access so that I can verify that things are working.

> One more question (I know, I'm a pest): Are there some datasets that cannot
> be accessed/stored this way (i.e. some data sets can't use the CURDAY 
> "replaceable"
> to show the current day, but keep older data in separate directories) The 
> reason I
> ask, is we're looking into storing some data offline (tape,DVD,etc...) and I 
> was
> wondering how we could store/restore this data as needed. I've made an entire 
> copy
> of our /data/ldm/mcidas directory. Before I tar/zip up these files, I thought 
> I'd ask
> you.

The \CURDAY construct was developed for realtime datasets -- \CURDAY gets 
expanded
to the current day (UTC) as determined by the system clock on the ADDE server.

SSEC added an undocumented feature that speeds access to archived datasets.  
Again,
the notion is that some portion of the fully qualified pathname (directory + 
file name)
contains a date.  The ADDE dataset defintion has an undocumented feature:  if
the keyword sequence RT=A is included in a dataset definition for image datasets
composed of AREA format files, then the data request is required to include
the keyword DAY= and the value of DAY= is used as a "replaceable".  SSEC added
this feature for their Data Center operations that serves huge volumes of 
archive
data.

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: TTR-495389
Department: Support McIDAS
Priority: Normal
Status: Closed