[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDD #TTR-495389]: Slow access to local datasets through ADDE



Hi Samuel,

re:
> Do I still need to put the 'CURDAY' option into the dsserve.k command string?
> (e.g. dsserve.k ADD RTIMAGES/ANTARCTIC AREA TYPE=IMAGE 
> DIRFILE=/data/ldm/mcidas/images/sat/ANTARCTIC/4km/IR/\\CURDAY/IR_* 
> \"Antarctic IR Composite)

The \CURDAY "replaceable" gets replaced by the CCYYMMDD value for the current 
day (UTC) in the
DIRFILE= regular expression that defines a datasets content.  This replacement 
occurs on the
server side when the ADDE request is received.  If you want to only access data 
from the current day, then
you would use \CURDAY.  If you want to access data from all days, you would use 
'*'.  Remember,
the discussion of \CURDAY arose from my comment that you would likely want to 
serve data
from the current day for the very numerous GINIEAST/GE1KVIS images.  This way, 
instead of having
to read and sort 'n*m' (number of days of data kept for a type of image TIMES 
number of images per day)
files, the ADDE server would only have to read 'n' files.  This becomes very 
significant when one tries
to keep a number of days of GINIEAST/GINIWEST Visible images online since there 
are 4 of each of
these each hour.

> If so, will it work this time?

It should, but...  I just tried creating an ADDE dataset for the Antarctic IR 
images using
the \CURDAY construct, and it didn't work.  This means that the mod I made to 
support \CURDAY
is not being used by the ADDE server for images in AREA format.  Rats!  I 
apologize for this
oversight on my part!!

The \CURDAY construct does work for datasets of images of type GINI (images in 
the GINIEAST,
GINIWEST, GINICOMP, and NEXRCOMP datasets) and images of type NEXR (images in 
the RTNEXRAD
dataset), so I suggest that you use it in those first.  I will work on the mods 
needed to add
\CURDAY support to datasets of type AREA and make an addendum.

Here is an example I just ran on your machine that uses three different 
"replaceables" to define
a dataset for NEXRAD Level III data:

<as 'mcidas' on your machine>
cd $MCDATA
dsserve.k ADD CURNIDS/N0R NEXR TYPE=IMAGE 
DIRFILE=/data/ldm/mcidas/images/NIDS/\\ID/\\TYPE/\\TYPE_\\CURDAY_\* \"Test of 
CURDAY for NIDS

The "replaceables" used are:

\ID     -> NEXRAD Level III station ID (e.g., FTG, DIX, PBZ, etc.)
\TYPE   -> NEXRAD Level III product type (e.g., N0R, N0S, etc.)
\CURDAY -> current data represented as CCYYMMDD

After making the CRUNIDS/N0R dataset definition on your machine, I am able to 
see the just the
current day's NEXRAD Level III base reflectivity (N0R) images from the station 
FTG:

-bash-2.05b$ imglist.k CURNIDS/N0R.ALL ID=FTG

Image file directory listing for:CURNIDS/N0R
 Pos Satellite/         Date       Time      Center   Band(s)
     sensor                                 Lat  Lon
 --- -------------  ------------  --------  ---- ---- ------------
   1  RADAR          7 FEB 08038  00:06:00     FTG    1
   2  RADAR          7 FEB 08038  00:16:00     FTG    1
   3  RADAR          7 FEB 08038  00:25:00     FTG    1
   4  RADAR          7 FEB 08038  00:35:00     FTG    1
   5  RADAR          7 FEB 08038  00:45:00     FTG    1
   6  RADAR          7 FEB 08038  00:55:00     FTG    1
   7  RADAR          7 FEB 08038  01:04:00     FTG    1
   8  RADAR          7 FEB 08038  01:14:00     FTG    1
   9  RADAR          7 FEB 08038  01:24:00     FTG    1
  10  RADAR          7 FEB 08038  01:33:00     FTG    1
  11  RADAR          7 FEB 08038  01:43:00     FTG    1
  12  RADAR          7 FEB 08038  01:53:00     FTG    1
  13  RADAR          7 FEB 08038  02:22:00     FTG    1
  14  RADAR          7 FEB 08038  03:30:00     FTG    1
  15  RADAR          7 FEB 08038  04:39:00     FTG    1
  16  RADAR          7 FEB 08038  06:37:00     FTG    1
  17  RADAR          7 FEB 08038  07:25:00     FTG    1
  18  RADAR          7 FEB 08038  07:35:00     FTG    1
  19  RADAR          7 FEB 08038  10:50:00     FTG    1
  20  RADAR          7 FEB 08038  11:48:00     FTG    1
  21  RADAR          7 FEB 08038  11:58:00     FTG    1
  22  RADAR          7 FEB 08038  12:08:00     FTG    1
  23  RADAR          7 FEB 08038  12:18:00     FTG    1
  24  RADAR          7 FEB 08038  12:27:00     FTG    1
  25  RADAR          7 FEB 08038  12:37:00     FTG    1
  26  RADAR          7 FEB 08038  12:47:00     FTG    1
  27  RADAR          7 FEB 08038  12:57:00     FTG    1
  28  RADAR          7 FEB 08038  13:06:00     FTG    1
  29  RADAR          7 FEB 08038  13:16:00     FTG    1
  30  RADAR          7 FEB 08038  13:26:00     FTG    1
  31  RADAR          7 FEB 08038  13:36:00     FTG    1
  32  RADAR          7 FEB 08038  15:34:00     FTG    1
  33  RADAR          7 FEB 08038  17:50:00     FTG    1
  34  RADAR          7 FEB 08038  18:00:00     FTG    1
  35  RADAR          7 FEB 08038  18:10:00     FTG    1
  36  RADAR          7 FEB 08038  18:19:00     FTG    1
  37  RADAR          7 FEB 08038  18:29:00     FTG    1
  38  RADAR          7 FEB 08038  18:39:00     FTG    1
  39  RADAR          7 FEB 08038  18:49:00     FTG    1
  40  RADAR          7 FEB 08038  18:58:00     FTG    1
  41  RADAR          7 FEB 08038  19:08:00     FTG    1
  42  RADAR          7 FEB 08038  19:18:00     FTG    1
  43  RADAR          7 FEB 08038  19:28:00     FTG    1
  44  RADAR          7 FEB 08038  19:37:00     FTG    1
  45  RADAR          7 FEB 08038  19:47:00     FTG    1
  46  RADAR          7 FEB 08038  19:57:00     FTG    1
  47  RADAR          7 FEB 08038  20:07:00     FTG    1
  48  RADAR          7 FEB 08038  20:16:00     FTG    1
  49  RADAR          7 FEB 08038  20:26:00     FTG    1
  50  RADAR          7 FEB 08038  20:36:00     FTG    1
  51  RADAR          7 FEB 08038  20:46:00     FTG    1
imglist.k: done

To demonstrate that it really is accessing a subset of the files, note the 
number of
days of images in the directory containing FTG N0R images:

-bash-2.05b$ ls /data/ldm/mcidas/images/NIDS/FTG/N0R
N0R_20080120_1640  N0R_20080125_0754  N0R_20080130_0121  N0R_20080204_0002
N0R_20080120_1650  N0R_20080125_0931  N0R_20080130_0131  N0R_20080204_0012
N0R_20080120_1709  N0R_20080125_1021  N0R_20080130_0141  N0R_20080204_0022
N0R_20080120_1719  N0R_20080125_1041  N0R_20080130_0150  N0R_20080204_0031
N0R_20080120_1729  N0R_20080125_1050  N0R_20080130_0200  N0R_20080204_0041
N0R_20080120_1738  N0R_20080125_1208  N0R_20080130_0210  N0R_20080204_0051
 ...
N0R_20080125_0350  N0R_20080130_0032  N0R_20080203_2254  N0R_20080207_2036
N0R_20080125_0410  N0R_20080130_0042  N0R_20080203_2304  N0R_20080207_2046
N0R_20080125_0518  N0R_20080130_0052  N0R_20080203_2333
N0R_20080125_0537  N0R_20080130_0102  N0R_20080203_2343
N0R_20080125_0547  N0R_20080130_0111  N0R_20080203_2352

\CURDAY, like all "replaceables" can be used anyplace in the DIRFILE= regular 
expression
where part of the fully qualified pathname is the date in CCYYMMDD format.  
This means
that you really don't have to organize images into daily directories... but 
experience
has shown that it is useful to do so.

Again, I apologize for the \CURDAY construct not working (as I had thought that 
it
did) for images of type AREA (e.g., UNIWISC images; the ones organized into the
RTIMAGES and CIMSS datasets).  I will find and fix the problem and make an 
addendum.

> Will scourBYnumber be able to identify and remove old directores? (e.g. 
> 20080206 should be deleted in March)

No.  You will need to switch to use of scourBYday.  scourBYday can be found in 
the same
directory as scourBYnumber on your machine: ~ldm/util.  The directory specified 
on the
scourBYday invocation command line will be the directory down to, but not 
including
the various days.  For instance, the invocation for scourBYday that keeps 5 days
of the Antarctic IR images would be:

scourBYday /data/ldm/mcidas/images/sat/ANTARCTIC/4km/IR  5


> P.S. I'm in the process of moving our NNEXRAD (NIDS) data to /usr/data (a 
> different file system).
> This should free up some disk I/O.

Excellent.  This should really help your I/O bottleneck as there are on the 
order of 17,500
NNEXRAD (NIDS) files received every hour!

> We are still in the process of ordering a computer with more RAM, and CALU 
> would like to utilize
> bandwidth to the fullest (even if it isn't always stored)

Very good.

> P.P.S.  I've noticed Shu pulls down NEXRAD2 data, but it doesn't seem to be 
> in the ADDE set, or on file.
> Am I missing something?

No, you are not missing anything.  McIDAS has no ADDE server for NEXRAD Level 2 
data.  The
IDV, however, can use the NEXRAD Level 2 data directly, so you may still want 
to ingest
the images and then make them available to machines where the IDV is used (NFS 
mount, samba, etc.).

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: TTR-495389
Department: Support McIDAS
Priority: Normal
Status: Closed