Karen,
We do the same for our data aging . . . with one exception:
nice find ${MOUNT} -mount -type f -mmin ${CUTOFF) -exec rm -f {} \;
Where CUTOFF is a "+" value. The "nice" can be crucial with NEXRAD data, as
find can sometimes monopolize processing at the expense of more valuable
processing.
--
Stonie R. Cooper
Planetary Data, Incorporated
On Wednesday 05 May 2004 05:06 pm, Karen Cooper wrote:
> FYI: regarding the scour script.
>
> On linux a simple modification to the scour script can allow you to
> remove files on the order of minutes (instead of the default days). It
> also works on Solaris if you install gfind and use it as opposed to the
> Solaris find command.
>
> The change is:
>
> original:
> find . -type f -mtime +$FINDAGE -name "$pattern" -exec rm -f {} \;) \
>
> new:
> find . -type f -mmin +$FINDAGE -name "$pattern" -exec rm -f {} \;) \
>
> The just modify your scour settings to be minutes not days.
>
> This has worked very well on many of my systems where I ony keep 1-4
> hours of data at a time.
>From owner-ldm-users@xxxxxxxxxxxxxxxx 05 2004 May -0600 11:41:13
Date: 05 May 2004 11:41:13 -0600
From: Steve Chiswell <chiz@xxxxxxxxxxxxxxxx>
In-Reply-To: <002a01c432c3$ac7692c0$8950ae80@xxxxxxxxxxx>
To: David Wojtowicz <davidw@xxxxxxxx>
Subject: Re: CRAFT data hardware
Received: (from majordo@localhost)
by unidata.ucar.edu (UCAR/Unidata) id i45HfLuq009701
for ldm-users-out; Wed, 5 May 2004 11:41:21 -0600 (MDT)
Received: from flip.unidata.ucar.edu (flip.unidata.ucar.edu [128.117.140.85])
by unidata.ucar.edu (UCAR/Unidata) with ESMTP id i45HfEtK009676;
Wed, 5 May 2004 11:41:14 -0600 (MDT)
Keywords: 200405051741.i45HfEtK009676
Received: (from chiz@localhost)
by flip.unidata.ucar.edu (UCAR/8.12.5/Submit) id i45HfEQd2120448;
Wed, 5 May 2004 11:41:14 -0600 (MDT)
X-Authentication-Warning: flip.unidata.ucar.edu: chiz set sender to
chiz@xxxxxxxxxxxxxxxx using -f
Reply-To: chiz@xxxxxxxx
Cc: ldm-users@xxxxxxxxxxxxxxxx,
GEMPAK support <support-gempak@xxxxxxxxxxxxxxxx>
References:
<69D72311B46FD4119EF100D0B77CF7DA0F08F4A8@xxxxxxxxxxxxxxxxxxxxxx>
<002a01c432c3$ac7692c0$8950ae80@xxxxxxxxxxx>
Content-Type: text/plain; charset=ISO8859-1
Content-Transfer-Encoding: 7bit
Organization: Unidata
Message-Id: <1083778873.2065374.16.camel@xxxxxxxxxxxxxxxxxxxxx>
Mime-Version: 1.0
X-Mailer: Ximian Evolution 1.2.0
Sender: owner-ldm-users@xxxxxxxxxxxxxxxx
Precedence: bulk
David,
IDV currently uses the uncompressed files (here at UPC, the ones that
dcnexr2 uncompresses).
Dcnexr2 for GEMPAk accomplishes 2 tasks:
1) uncompresses the data stream
2) adds the station ID to the data file header (eg the -s option)
so that programs can determine where to plot the data based on the data,
rather than having to use the file name. This adds the station ID to
bytes 21-24 of the data header for the "ARCHIVE2." products, which is
what is done in the build5.0 data with header "AR2V0001.
#2 above is needed for the pre-5.0 RDA build sites. Once all of the
radar sites are upgraded to 5.0, this won't be necessary, and therefore
the added step of uncompressing with dcnexr2 would be superfluous.
Dcnexr2 is a bridge until build 5.0 is in place.
I will be adding the necessary uncompression to the GEMPAK routines,
but until all the sites are at build 5.0, the station ID will still have
to be provided either in the data header, or by the file template.
Steve Chiswell
On Wed, 2004-05-05 at 11:09, David Wojtowicz wrote:
> > Thanks. GEMPAK doesn't have decompress built in so we would have to at
> > least decompress and keep about 12 hours or so on disk uncompressed.
> >
>
> I think it'd be great if GEMPAK had the decompress built in. Does IDV
> handle the compressed files? The problem is the uncompressing does take up
> a huge amount of disk space, and is extremely CPU intensive, so you're
> doing all this work on data, that you probably are only going to look at 1%
> of.
>
> -----
> David Wojtowicz, Sr. Research Programmer, Sysadmin
> Dept of Atmospheric Sciences / Computer Services
> University of Illinois at Urbana-Champaign
> davidw@xxxxxxxx (217) 333-8390
