[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[McIDAS #XRR-599672]: FW: Please remove port 23 access from shu.cup.edu, and add ports 22 & 112.

Hi Samuel,

re: amount of data being kept on disk at CUP; is scouring being run out of cron
> Yes, I believe scour is being executed out of cron.

OK, just checking.

> It turns out that the queue size grows/shrinks from day to day.

The LDM queue size is fixed when it is created: there is no code in the LDM to
shrink or grow the queue.  There used to be a LONG time ago, but that was only
for IRIX machines.  We removed the dynamic queue size since it turned out to
be a bad idea (e.g., one could create a queue on a system with just enough
disk space for that sized queue; when the LDM tried to grow it, major problems
would be experienced).

> After I had reduced the scourBYnumber entries to only holding 64/48 entries, 
> and it
> seemed to shrink everything down to around 250G, (along with mcscour.sh 
> entries limited
> to 5).

Something is _really_ wrong here.  Keeping 64 of each type of image plus 5 MD 
files of
each type should only use a fraction of the disk space that you indicate is 
being
used on your machine (250 GB).  I am keeping 64 of each type of image, and I am
using about 6.2 GB of disk for all of them.

Since you are ingesting model data (GRIB and BUFR) and processing it with
the MySQL approach, it means that GRIB messages (GRIB1 and GRIB2) and BUFR 
messages
are being saved on your disk.  It is these files that I would guess are not 
being
scoured properly.

The routine that scours the GRIB and BUFR messages is 'xcdscour'.  It should be 
found
in the ~ldm/util directory, and it should be run out of a cron entry specifying
how many days of data to keep on disk.  For instance, the crontab entries that
I use to scour BUFR and GRIB data are:

10 19 * * * util/xcdscour BUFR 1 /machine/data/ldm/mcidas/bufr
20 19 * * * util/xcdscour GRIB 1 /machine/data/ldm/mcidas/grib

These entries say to keep 1 day of GRIB and BUFR files.

The disk space being used on my machine at 23 UTC is:

/machine/data/ldm/mcidas/grib% du -k .
27491812        .
/machine/data/ldm/mcidas/bufr% du -k .
2123896 .

This shows that I have about 27 MB of GRIB data and 2 GB of BUFR data for just
under 2 days for all HDS and NGRID data, or about 15 GB of data per day.

Have you checked to make sure that your GRIB and BUFR data are being scoured
correctly?

> Now I'm trying to get my disk utilization back up to around 500g, so I 
> increased
> my scourBYnumbers to 128/96, and the mscsour.sh entries up to 9, and I'm 
> waiting
> to see what'll happen.

First thing I would do is check to make sure that model data is being scoured
correctly.  I would CD to the directory where GRIB files are stored and make
sure that there are no old files.

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: XRR-599672
Department: Support McIDAS
Priority: Normal
Status: Closed