Donna,
Yours is zfs on centos 6.x or 7.x?
Neil
On Oct 28, 2015, at 3:22 PM, Donna Cote <d-cote@xxxxxxxx> wrote:
> Tom,
>
> We moved our storage LDM system to zfs (on the storage disks, I don't think
> the system filesystems changed) after experiencing problems with scour on xfs.
>
> When using xfs, we saw the load average regularly rise to 50, 60, once to 110
> (15-minute average over all CPUs). Even after the job of removing a day's
> worth of radar data (x days back) was moved to a bash script. The scours
> would start stacking (per Karen's definition) and I even created a bash
> script to kill all scour jobs every 3 hours.
>
> Now, with zfs, the load average rarely rises above 6 and the amount of data
> coming in just about equals the amount of old data we remove every day. The
> scour jobs have not done any stacking.
>
> Donna
> The Academy
> Texas A&M University
>
>
> On Wed, Oct 28, 2015 at 11:20 AM, Tom Yoksas <yoksas@xxxxxxxx> wrote:
> Hi Mike,
>
> On 10/28/2015 10:02 AM, Michael Dross wrote:
> Is there one type of filesystem that is more efficient at handling
> these types file large volumes of finds/creations/deletions?
>
> ie. EXT4, zfs.
>
> We (Unidata Program Center) have had good luck with ZFS on our
> Solaris 10 motherlode-class machines.
>
> It would be most interesting to hear other users experiences!
>
> Cheers,
>
> Tom
>
> On Oct 28, 2015, at 11:56 AM, Tom Yoksas <yoksas@xxxxxxxx> wrote:
>
> All,
>
> re: how to efficiently scour LDM-created data products from disk
>
> If one organizes one's data files into daily directories, the need to
> 'find' each file that should be scoured can be pared down to finding
> which directories+contents need to be scoured. This does not, however,
> get around the fact that deleting large numbers of files
> (e.g., millions for NEXRAD Level III files kept on disk for long
> periods of time) is a _very_ costly endeavor.
>
> Cheers,
>
> Tom
>
> On 10/28/2015 09:47 AM, Gerry Creager - NOAA Affiliate wrote:
> How often to you run your scour command? We've exercised scour pretty
> hard and were able to watch it keep up, if it were run frequently.
> There's a one-time penalty, if you've been running it, say, daily or
> less frequently, but if you run it hourly, it should keep up.
>
> gerry
>
> On Wed, Oct 28, 2015 at 10:39 AM, Steve Emmerson <emmerson@xxxxxxxx
> <mailto:emmerson@xxxxxxxx>> wrote:
>
> LDMers,
>
> On Wed, Oct 28, 2015 at 9:28 AM, Donna Cote <d-cote@xxxxxxxx
> <mailto:d-cote@xxxxxxxx>> wrote:
>
> I would like to ask for a more io-friendly scour utility.
>
>
> It's on my list (see <https://github.com/Unidata/LDM/issues/27>.
> Can't say when I'll get around to it, though.
>
> Regards,
> Steve Emmerson
>
> _______________________________________________
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx <mailto:ldm-users@xxxxxxxxxxxxxxxx>
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
> --
> Gerry Creager
> NSSL/CIMMS
> 405.325.6371
> ++++++++++++++++++++++
> “Big whorls have little whorls,
> That feed on their velocity;
> And little whorls have lesser whorls,
> And so on to viscosity.”
> Lewis Fry Richardson (1881-1953)
>
> _______________________________________________
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
> --
> +----------------------------------------------------------------------+
> * Tom Yoksas UCAR Unidata Program *
> * (303) 497-8642 (last resort) P.O. Box 3000 *
> * yoksas@xxxxxxxx Boulder, CO 80307 *
> * Unidata WWW Service http://www.unidata.ucar.edu/ *
> +----------------------------------------------------------------------+
>
> _______________________________________________
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
> --
> +----------------------------------------------------------------------+
> * Tom Yoksas UCAR Unidata Program *
> * (303) 497-8642 (last resort) P.O. Box 3000 *
> * yoksas@xxxxxxxx Boulder, CO 80307 *
> * Unidata WWW Service http://www.unidata.ucar.edu/ *
> +----------------------------------------------------------------------+
>
> _______________________________________________
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
> _______________________________________________
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
