Is there one type of filesystem that is more efficient at handling these types
file large volumes of finds/creations/deletions?
ie. EXT4, zfs.
- Mike
> On Oct 28, 2015, at 11:56 AM, Tom Yoksas <yoksas@xxxxxxxx> wrote:
>
> All,
>
> re: how to efficiently scour LDM-created data products from disk
>
> If one organizes one's data files into daily directories, the need to
> 'find' each file that should be scoured can be pared down to finding
> which directories+contents need to be scoured. This does not, however,
> get around the fact that deleting large numbers of files
> (e.g., millions for NEXRAD Level III files kept on disk for long
> periods of time) is a _very_ costly endeavor.
>
> Cheers,
>
> Tom
>
>> On 10/28/2015 09:47 AM, Gerry Creager - NOAA Affiliate wrote:
>> How often to you run your scour command? We've exercised scour pretty
>> hard and were able to watch it keep up, if it were run frequently.
>> There's a one-time penalty, if you've been running it, say, daily or
>> less frequently, but if you run it hourly, it should keep up.
>>
>> gerry
>>
>> On Wed, Oct 28, 2015 at 10:39 AM, Steve Emmerson <emmerson@xxxxxxxx
>> <mailto:emmerson@xxxxxxxx>> wrote:
>>
>> LDMers,
>>
>> On Wed, Oct 28, 2015 at 9:28 AM, Donna Cote <d-cote@xxxxxxxx
>> <mailto:d-cote@xxxxxxxx>> wrote:
>>
>> I would like to ask for a more io-friendly scour utility.
>>
>>
>> It's on my list (see <https://github.com/Unidata/LDM/issues/27>.
>> Can't say when I'll get around to it, though.
>>
>> Regards,
>> Steve Emmerson
>>
>> _______________________________________________
>> ldm-users mailing list
>> ldm-users@xxxxxxxxxxxxxxxx <mailto:ldm-users@xxxxxxxxxxxxxxxx>
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>> --
>> Gerry Creager
>> NSSL/CIMMS
>> 405.325.6371
>> ++++++++++++++++++++++
>> “Big whorls have little whorls,
>> That feed on their velocity;
>> And little whorls have lesser whorls,
>> And so on to viscosity.”
>> Lewis Fry Richardson (1881-1953)
>>
>> _______________________________________________
>> ldm-users mailing list
>> ldm-users@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>
> --
> +----------------------------------------------------------------------+
> * Tom Yoksas UCAR Unidata Program *
> * (303) 497-8642 (last resort) P.O. Box 3000 *
> * yoksas@xxxxxxxx Boulder, CO 80307 *
> * Unidata WWW Service http://www.unidata.ucar.edu/ *
> +----------------------------------------------------------------------+
>
> _______________________________________________
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
