Karen, the C program you
guys
wrote sound
s
really neat.
What kind of filesystem are you using? xfs? ext4? zfs?
What I have heard is that a remove done on xfs can take a really long time
compared to a remove done on zfs.
We switched the hard drives on our LDM storage server to zfs and find that
a scour every hour works well.
We can't do the same on our EDEX / AWIPS II server and would just like to
get it running smoothly. It seems I'm in constant nanny mode with it.
Donna
On Oct 28, 2015 11:45 AM, "Karen Cooper - NOAA Affiliate" <
karen.cooper@xxxxxxxx> wrote:
> Here at NSSL we wrote a cleanup program because at times scour did not
> keep up with our data.
>
> To be fair, we were using scour to clean out not only our incoming data,
> but also a lot of products (MRMS) we were creating from that data.
>
> We had originally tried Gerry's suggested method of running scour more
> frequently, which worked as long is it did not "stack". Stacking meaning
> that a new scour would begin before the last was complete. When that
> happened they would typically continue to stack and spiral out of control .
>
> So we wrote our own that would do single recursive searches of our
> directories rather than having to do multiple passes on directories,
> looking for different patterns. It runs as a daemon, so when it is done,
> it sleeps for a configurable amount of time and then runs again.
>
> We could probably have just been more careful with crafting our patterns
> in scour.conf, but since we used it so heavily the C program seemed the way
> to go and it has worked very well for us.
>
> On Wed, Oct 28, 2015 at 11:31 AM, Smith, Neil R <n-smith2@xxxxxxxxxxxxx>
> wrote:
>
>> Good question - isn’t it the same I/O process, just a matter of when?
>>
>> I'm wondering if the programming overhead of a file modification time
>> parameter test conducted by the ‘find’ command makes it worth avoiding.
>>
>> Gerry Creager is suggesting that a solution is to just increase the
>> frequency of running a find command (scour). It’s worth a try, but I don’t
>> see how that escapes the convergence of time to eval delete qualifications
>> inode list expansion rate.
>>
>> But if that’s a working solution, then — in the limit — so is a file
>> delete action executed coincident with a new file write.
>>
>> I interpret Tom Yoksas as saying to fail over to a different file system
>> catalogue operation - removal of groups of inodes instead of removal of
>> each inode separately.
>>
>> Good idea. And the one Donna Cote settled on. Which will require me to
>> adjust all my friggin gempak GUI configs and scripted real-time graphics
>> product generation.
>>
>> Which I'm getting paid to do ...
>>
>> -Neil
>>
>> > On Oct 28, 2015, at 10:28 AM, Donna Cote <d-cote@xxxxxxxx> wrote:
>> >
>> > Neil, while it is possible to use the EXEC argument, given the
>> filename, to create the rm command line statment (on that filename minus x
>> number of days), I don't see how this would make less removes than you are
>> already doing.
>> >
>> > I would like to ask for a more io-friendly scour utility.
>> >
>> > Donna
>> > Texas A&M University
>> > The Academy
>> >
>> >
>> > On Wed, Oct 28, 2015 at 10:17 AM, Smith, Neil R <n-smith2@xxxxxxxxxxxxx>
>> wrote:
>> > Scouring 10 days of online, or even nearline, NIDS and NEXRCOMP being
>> held for a teaching environment is a big headache.
>> > Donna Cote has recently polled the community about this issue.
>> >
>> > At this file count, crawling thru the millions of inodes with a
>> scheduled find command will not keep up with the accumulation rate.
>> > (sounds like the intersection of two curves, beyond which there is no
>> return; would anyone care to derive that solution?)
>> >
>> > This leads me to wonder if one can construct a pqact entry that will do
>> the FILE or STDIOFILE action on the incoming product plus remove the same
>> product of the same date minus a desired day count. Or the remove action,
>> -exec, could be a separate pqact entry on the same product regular
>> expression minus the day count, just so it’s done at the time of the new
>> product arrival. Is there a clever way to manipulate the syntax of
>> temporal subexpression replacement described in the pqact man pages to
>> generate the desired filename to remove?
>> >
>> > Would that even solve the problem?
>> >
>> > Or is there a pqact solution? Are you just left with scripting OS
>> commands (find; eg. scour)?
>> >
>> > -Neil
>> >
>> > --
>> > Neil R. Smith, Senior IT Specialist II neils@xxxxxxxx
>> > Atmospheric Sciences, Texas A&M Univ. 979/845-6272
>> > --
>> >
>> > _______________________________________________
>> > ldm-users mailing list
>> > ldm-users@xxxxxxxxxxxxxxxx
>> > For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>> >
>>
>> _______________________________________________
>> ldm-users mailing list
>> ldm-users@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>>
>
>
>
> --
> “The presence of those seeking the truth is infinitely to be preferred to
> the presence of those who think they’ve found it."
> -- Terry Prachett
>
> -------------------------------------------
> Karen.Cooper@xxxxxxxx
>
> Phone#: 405-325-6456
> Cell: 405-834-8559
> National Severe Storms Laboratory
>
>
