[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[McIDAS #ILV-152686]: What is nexraget?

Hi Mike,

re:
> If I understand you correctly, you're saying that how the directory
> structure is designed could impact the overall performance?  Interesting.

Yes, and this is true for a large variety of applications, not just McIDAS.
The approach of McIDAS ADDE servers is to crack every data file that is
part of a dataset and extract the information on product type, date, time
etc. from the file contents.  This means that if one creates a dataset
definition that contains a very large number of elements, then it will take
a long time for the server(s) to read through the files in order to choose
the dataset element (file) that satisfies the request by the user.  What
I was not saying is that I would expect that the process to be flaky
because of this; I would expect it to be slower.

re:
> For ours we save by [station]/[type]/ and save with a date_time filename.
> We process level 3 data in real time with Mcidas as it comes over LDM for
> the sake of making single-site images for -most- products.  As the volume
> of data has been increasing we've noticed the workload increasing as well.

Yup, there are more files to be looked through (opened and their header's
read), and this will take more system resources.

re:
> Tricks like this would be helpful.

For image data served off of our "motherlode class" servers, we (I) tend
to keep "current" datasets down to a minimum number of files.  For instance,
for the datasets created from IDD UNIWISC products, I keep 96 images
in the "current" set since that is what most people seem to want to
access (actually, most people request only the newest image in any
dataset).  I came up with keeping 96 after seeing what users were
requesting, and then going out a couple/three standard deviations
and then re-adjusted when I got user input on wanting more images in
the current sets.

re: 
> Thanks,

No worries.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: ILV-152686
Department: Support McIDAS
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.