We've mostly used it on ext2 and ext3. We have also used it on zfs.
I don't think we have any xfs filesystems that we have run it on.
On Wed, Oct 28, 2015 at 2:35 PM, Donna Cote <d-cote@xxxxxxxx> wrote:
> Karen, the C program you
> guys
> wrote sound
> s
> really neat.
> What kind of filesystem are you using? xfs? ext4? zfs?
>
> What I have heard is that a remove done on xfs can take a really long
> time compared to a remove done on zfs.
>
> We switched the hard drives on our LDM storage server to zfs and find
> that a scour every hour works well.
>
> We can't do the same on our EDEX / AWIPS II server and would just like to
> get it running smoothly. It seems I'm in constant nanny mode with it.
>
> Donna
>
>
> On Oct 28, 2015 11:45 AM, "Karen Cooper - NOAA Affiliate" <
> karen.cooper@xxxxxxxx> wrote:
>
>> Here at NSSL we wrote a cleanup program because at times scour did not
>> keep up with our data.
>>
>> To be fair, we were using scour to clean out not only our incoming data,
>> but also a lot of products (MRMS) we were creating from that data.
>>
>> We had originally tried Gerry's suggested method of running scour more
>> frequently, which worked as long is it did not "stack". Stacking meaning
>> that a new scour would begin before the last was complete. When that
>> happened they would typically continue to stack and spiral out of control.
>>
>> So we wrote our own that would do single recursive searches of our
>> directories rather than having to do multiple passes on directories,
>> looking for different patterns. It runs as a daemon, so when it is done,
>> it sleeps for a configurable amount of time and then runs again.
>>
>> We could probably have just been more careful with crafting our patterns
>> in scour.conf, but since we used it so heavily the C program seemed the way
>> to go and it has worked very well for us.
>>
>> On Wed, Oct 28, 2015 at 11:31 AM, Smith, Neil R <n-smith2@xxxxxxxxxxxxx>
>> wrote:
>>
>>> Good question - isn’t it the same I/O process, just a matter of when?
>>>
>>> I'm wondering if the programming overhead of a file modification time
>>> parameter test conducted by the ‘find’ command makes it worth avoiding.
>>>
>>> Gerry Creager is suggesting that a solution is to just increase the
>>> frequency of running a find command (scour). It’s worth a try, but I don’t
>>> see how that escapes the convergence of time to eval delete qualifications
>>> inode list expansion rate.
>>>
>>> But if that’s a working solution, then — in the limit — so is a file
>>> delete action executed coincident with a new file write.
>>>
>>> I interpret Tom Yoksas as saying to fail over to a different file system
>>> catalogue operation - removal of groups of inodes instead of removal of
>>> each inode separately.
>>>
>>> Good idea. And the one Donna Cote settled on. Which will require me to
>>> adjust all my friggin gempak GUI configs and scripted real-time graphics
>>> product generation.
>>>
>>> Which I'm getting paid to do ...
>>>
>>> -Neil
>>>
>>> > On Oct 28, 2015, at 10:28 AM, Donna Cote <d-cote@xxxxxxxx> wrote:
>>> >
>>> > Neil, while it is possible to use the EXEC argument, given the
>>> filename, to create the rm command line statment (on that filename minus x
>>> number of days), I don't see how this would make less removes than you are
>>> already doing.
>>> >
>>> > I would like to ask for a more io-friendly scour utility.
>>> >
>>> > Donna
>>> > Texas A&M University
>>> > The Academy
>>> >
>>> >
>>> > On Wed, Oct 28, 2015 at 10:17 AM, Smith, Neil R <
>>> n-smith2@xxxxxxxxxxxxx> wrote:
>>> > Scouring 10 days of online, or even nearline, NIDS and NEXRCOMP being
>>> held for a teaching environment is a big headache.
>>> > Donna Cote has recently polled the community about this issue.
>>> >
>>> > At this file count, crawling thru the millions of inodes with a
>>> scheduled find command will not keep up with the accumulation rate.
>>> > (sounds like the intersection of two curves, beyond which there is no
>>> return; would anyone care to derive that solution?)
>>> >
>>> > This leads me to wonder if one can construct a pqact entry that will
>>> do the FILE or STDIOFILE action on the incoming product plus remove the
>>> same product of the same date minus a desired day count. Or the remove
>>> action, -exec, could be a separate pqact entry on the same product regular
>>> expression minus the day count, just so it’s done at the time of the new
>>> product arrival. Is there a clever way to manipulate the syntax of
>>> temporal subexpression replacement described in the pqact man pages to
>>> generate the desired filename to remove?
>>> >
>>> > Would that even solve the problem?
>>> >
>>> > Or is there a pqact solution? Are you just left with scripting OS
>>> commands (find; eg. scour)?
>>> >
>>> > -Neil
>>> >
>>> > --
>>> > Neil R. Smith, Senior IT Specialist II neils@xxxxxxxx
>>> > Atmospheric Sciences, Texas A&M Univ. 979/845-6272
>>> > --
>>> >
>>> > _______________________________________________
>>> > ldm-users mailing list
>>> > ldm-users@xxxxxxxxxxxxxxxx
>>> > For list information or to unsubscribe, visit:
>>> http://www.unidata.ucar.edu/mailing_lists/
>>> >
>>>
>>> _______________________________________________
>>> ldm-users mailing list
>>> ldm-users@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe, visit:
>>> http://www.unidata.ucar.edu/mailing_lists/
>>>
>>
>>
>>
>> --
>> “The presence of those seeking the truth is infinitely to be preferred to
>> the presence of those who think they’ve found it."
>> -- Terry Prachett
>>
>> -------------------------------------------
>> Karen.Cooper@xxxxxxxx
>>
>> Phone#: 405-325-6456
>> Cell: 405-834-8559
>> National Severe Storms Laboratory
>>
>>
--
“The presence of those seeking the truth is infinitely to be preferred to
the presence of those who think they’ve found it."
-- Terry Prachett
-------------------------------------------
Karen.Cooper@xxxxxxxx
Phone#: 405-325-6456
Cell: 405-834-8559
National Severe Storms Laboratory
