Here at NSSL we wrote a cleanup program because at times scour did not keep
up with our data.
To be fair, we were using scour to clean out not only our incoming data,
but also a lot of products (MRMS) we were creating from that data.
We had originally tried Gerry's suggested method of running scour more
frequently, which worked as long is it did not "stack". Stacking meaning
that a new scour would begin before the last was complete. When that
happened they would typically continue to stack and spiral out of control.
So we wrote our own that would do single recursive searches of our
directories rather than having to do multiple passes on directories,
looking for different patterns. It runs as a daemon, so when it is done,
it sleeps for a configurable amount of time and then runs again.
We could probably have just been more careful with crafting our patterns in
scour.conf, but since we used it so heavily the C program seemed the way to
go and it has worked very well for us.
On Wed, Oct 28, 2015 at 11:31 AM, Smith, Neil R <n-smith2@xxxxxxxxxxxxx>
wrote:
> Good question - isn’t it the same I/O process, just a matter of when?
>
> I'm wondering if the programming overhead of a file modification time
> parameter test conducted by the ‘find’ command makes it worth avoiding.
>
> Gerry Creager is suggesting that a solution is to just increase the
> frequency of running a find command (scour). It’s worth a try, but I don’t
> see how that escapes the convergence of time to eval delete qualifications
> inode list expansion rate.
>
> But if that’s a working solution, then — in the limit — so is a file
> delete action executed coincident with a new file write.
>
> I interpret Tom Yoksas as saying to fail over to a different file system
> catalogue operation - removal of groups of inodes instead of removal of
> each inode separately.
>
> Good idea. And the one Donna Cote settled on. Which will require me to
> adjust all my friggin gempak GUI configs and scripted real-time graphics
> product generation.
>
> Which I'm getting paid to do ...
>
> -Neil
>
> > On Oct 28, 2015, at 10:28 AM, Donna Cote <d-cote@xxxxxxxx> wrote:
> >
> > Neil, while it is possible to use the EXEC argument, given the filename,
> to create the rm command line statment (on that filename minus x number of
> days), I don't see how this would make less removes than you are already
> doing.
> >
> > I would like to ask for a more io-friendly scour utility.
> >
> > Donna
> > Texas A&M University
> > The Academy
> >
> >
> > On Wed, Oct 28, 2015 at 10:17 AM, Smith, Neil R <n-smith2@xxxxxxxxxxxxx>
> wrote:
> > Scouring 10 days of online, or even nearline, NIDS and NEXRCOMP being
> held for a teaching environment is a big headache.
> > Donna Cote has recently polled the community about this issue.
> >
> > At this file count, crawling thru the millions of inodes with a
> scheduled find command will not keep up with the accumulation rate.
> > (sounds like the intersection of two curves, beyond which there is no
> return; would anyone care to derive that solution?)
> >
> > This leads me to wonder if one can construct a pqact entry that will do
> the FILE or STDIOFILE action on the incoming product plus remove the same
> product of the same date minus a desired day count. Or the remove action,
> -exec, could be a separate pqact entry on the same product regular
> expression minus the day count, just so it’s done at the time of the new
> product arrival. Is there a clever way to manipulate the syntax of
> temporal subexpression replacement described in the pqact man pages to
> generate the desired filename to remove?
> >
> > Would that even solve the problem?
> >
> > Or is there a pqact solution? Are you just left with scripting OS
> commands (find; eg. scour)?
> >
> > -Neil
> >
> > --
> > Neil R. Smith, Senior IT Specialist II neils@xxxxxxxx
> > Atmospheric Sciences, Texas A&M Univ. 979/845-6272
> > --
> >
> > _______________________________________________
> > ldm-users mailing list
> > ldm-users@xxxxxxxxxxxxxxxx
> > For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
> >
>
> _______________________________________________
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
--
“The presence of those seeking the truth is infinitely to be preferred to
the presence of those who think they’ve found it."
-- Terry Prachett
-------------------------------------------
Karen.Cooper@xxxxxxxx
Phone#: 405-325-6456
Cell: 405-834-8559
National Severe Storms Laboratory
