Load is back to the neighborhood of 25-30 again. Attached is output from
iostat and sar.
I've also run htop as suggested. It typically shows cpu usage varying from
little more than 5% on most of the 8 processors to occasional periods when all
eight are between 70-85%.
I'm not sure what to look for in terms of slow disk writes/reads being the
culprit. The fourth internal disk in the system is where all the
EDEX/LDM-related data is filed.
Thanks,
Kevin
_____________________________________________
Kevin Tyle, Systems Administrator
Dept. of Atmospheric & Environmental Sciences
University at Albany
Earth Science 235, 1400 Washington Avenue
Albany, NY 12222
Email: ktyle@xxxxxxxxxx
Phone: 518-442-4578
_____________________________________________
-----Original Message-----
From: daryl herzmann [mailto:akrherz@xxxxxxxxxxx]
Sent: Wednesday, October 23, 2013 2:01 PM
To: Tyle, Kevin R
Cc: awips2-users@xxxxxxxxxxxxxxxx
Subject: Re: [awips2-users] CPU load on EDEX/LDM server
Kevin,
It would be good to interogate system monitoring tools like 'sar' to see what
it is logging for CPU vs IO wait percentages. In general, high CPU load is
typically tied to processes / threads waiting for IO to occur. If your disk is
unable to keep up with raw IO or intensive metadata operations, that could be
the trouble.
Running 'top' will also show you if your processes are chewing actual CPU or
the load average is artificially high with IO wait being to blame.
daryl
On Wed, 23 Oct 2013, Tyle, Kevin R wrote:
> A follow-up: the high CPU load was almost certainly linked to the ldm server
> that was running. Once it was shut down, load dropped to close to 0. Upon
> restart, load jumped back up to the vicinity of 20.00. I then shut down the
> LDM and purged the data directories (which were very very large since no
> scouring had been done). Restarting the LDM shows low system load so far. I
> will enable LDM scouring and keep an eye on things.
>
> _____________________________________________
> Kevin Tyle, Systems Administrator
> Dept. of Atmospheric & Environmental Sciences University at Albany
> Earth Science 235, 1400 Washington Avenue Albany, NY 12222
> Email: ktyle@xxxxxxxxxx<mailto:ktyle@xxxxxxxxxx>
> Phone: 518-442-4578
> _____________________________________________
>
> From: awips2-users-bounces@xxxxxxxxxxxxxxxx
> [mailto:awips2-users-bounces@xxxxxxxxxxxxxxxx] On Behalf Of Tyle,
> Kevin R
> Sent: Tuesday, October 22, 2013 11:00 AM
> To: awips2-users@xxxxxxxxxxxxxxxx
> Subject: [awips2-users] CPU load on EDEX/LDM server
>
> Hi,
>
> I am running an EDEX server, also running LDM, using the 13.2 version
> that Michael has made available to us. It is on a CentOS5 platform,
> 32-bit, 16 CPU, 12 GB RAM. I am not running CAVE on this machine.
>
> I regularly see an average load around 20.00, which seems excessive.
> Lots of java, httpd, postmaster processes owned by user awips.
>
> Does this load seem excessive? What do others see? If this load indeed
> seems excessive, any ideas to troubleshoot?
>
> Thanks,
>
> Kevin
>
> _____________________________________________
> Kevin Tyle, Systems Administrator
> Dept. of Atmospheric & Environmental Sciences University at Albany
> Earth Science 235, 1400 Washington Avenue Albany, NY 12222
> Email: ktyle@xxxxxxxxxx<mailto:ktyle@xxxxxxxxxx>
> Phone: 518-442-4578
> _____________________________________________
>
>
--
/**
* Daryl Herzmann
* Assistant Scientist -- Iowa Environmental Mesonet
* http://mesonet.agron.iastate.edu
*/
Linux 2.6.18-371.1.2.el5PAE (edex2.atmos.albany.edu) 10/24/2013
12:00:01 AM CPU %user %nice %system %iowait %steal
%idle
12:10:02 AM all 16.57 0.00 2.70 9.95 0.00
70.78
12:20:02 AM all 10.73 0.00 2.12 12.12 0.00
75.03
12:30:06 AM all 11.89 0.00 2.68 15.83 0.00
69.60
12:40:01 AM all 11.05 0.00 2.09 6.98 0.00
79.88
12:50:02 AM all 7.14 0.00 2.01 8.37 0.00
82.48
01:00:01 AM all 7.05 0.00 1.96 3.62 0.00
87.38
01:10:01 AM all 7.85 0.00 2.10 4.33 0.00
85.72
01:20:01 AM all 7.33 0.00 2.03 4.10 0.00
86.54
01:30:05 AM all 11.05 0.00 2.53 5.72 0.00
80.69
01:40:02 AM all 21.75 0.00 2.94 12.56 0.00
62.75
01:50:01 AM all 22.74 0.00 3.58 14.55 0.00
59.13
02:00:01 AM all 25.66 0.00 3.45 11.86 0.00
59.03
02:10:01 AM all 20.75 0.00 3.00 15.50 0.00
60.76
02:20:02 AM all 21.68 0.00 3.09 14.36 0.00
60.87
02:30:01 AM all 18.71 0.00 3.17 16.83 0.00
61.30
02:40:01 AM all 23.34 0.00 3.27 18.65 0.00
54.74
02:50:01 AM all 21.43 0.00 3.49 21.45 0.00
53.63
03:00:01 AM all 20.40 0.00 2.99 13.64 0.00
62.98
03:10:01 AM all 23.19 0.00 3.34 16.38 0.00
57.08
03:20:01 AM all 22.37 0.00 2.98 16.30 0.00
58.35
03:30:02 AM all 18.61 0.00 3.12 16.61 0.00
61.65
03:40:01 AM all 26.01 0.00 3.56 13.41 0.00
57.02
03:50:01 AM all 26.09 0.00 3.29 13.38 0.00
57.24
04:00:01 AM all 21.70 0.00 3.10 16.35 0.00
58.85
04:10:02 AM all 16.97 0.07 2.93 19.56 0.00
60.46
04:20:01 AM all 13.57 1.37 3.34 17.15 0.00
64.57
04:30:02 AM all 10.92 0.00 2.78 26.11 0.00
60.18
04:40:01 AM all 20.68 0.00 3.15 23.97 0.00
52.20
04:50:01 AM all 19.35 0.00 3.17 20.38 0.00
57.10
05:00:01 AM all 19.53 0.00 3.10 18.52 0.00
58.85
05:10:01 AM all 19.48 0.00 3.23 17.31 0.00
59.98
05:20:01 AM all 15.27 0.00 2.84 19.58 0.00
62.31
05:30:03 AM all 14.23 0.00 3.07 21.75 0.00
60.95
05:40:01 AM all 25.55 0.00 3.91 16.05 0.00
54.50
05:50:01 AM all 22.47 0.00 3.33 18.68 0.00
55.52
06:00:01 AM all 20.89 0.00 3.30 19.73 0.00
56.07
06:10:01 AM all 19.67 0.00 3.27 18.45 0.00
58.61
06:20:01 AM all 17.49 0.00 2.57 17.62 0.00
62.32
06:30:01 AM all 11.34 0.00 2.62 18.73 0.00
67.31
06:40:01 AM all 17.47 0.00 3.14 18.59 0.00
60.80
06:50:01 AM all 15.08 0.00 2.75 16.06 0.00
66.10
07:00:02 AM all 13.22 0.00 2.54 15.98 0.00
68.25
07:10:01 AM all 11.84 0.00 2.55 16.88 0.00
68.73
07:20:01 AM all 16.06 0.00 2.86 10.41 0.00
70.68
07:30:01 AM all 12.56 0.00 2.85 19.57 0.00
65.03
07:40:01 AM all 17.13 0.00 2.36 13.97 0.00
66.55
07:50:03 AM all 14.97 0.00 2.67 16.81 0.00
65.56
08:00:01 AM all 18.18 0.00 3.11 15.58 0.00
63.14
08:10:01 AM all 15.34 0.00 2.80 18.09 0.00
63.78
08:20:01 AM all 23.37 0.00 3.32 16.65 0.00
56.66
08:30:02 AM all 16.28 0.00 3.02 17.71 0.00
62.98
08:40:02 AM all 18.23 0.00 3.25 23.04 0.00
55.48
08:50:02 AM all 15.34 0.00 2.51 26.38 0.00
55.78
09:00:03 AM all 12.73 0.00 2.65 23.27 0.00
61.35
09:10:04 AM all 16.63 0.00 3.04 19.16 0.00
61.17
09:20:01 AM all 18.61 0.00 2.97 15.44 0.00
62.97
09:30:01 AM all 19.84 0.00 3.04 17.95 0.00
59.17
09:40:01 AM all 18.72 0.00 2.82 17.52 0.00
60.95
09:50:02 AM all 20.68 0.00 3.16 11.27 0.00
64.89
10:00:01 AM all 23.63 0.00 3.38 14.12 0.00
58.88
10:10:01 AM all 20.50 0.00 3.14 16.82 0.00
59.54
10:20:01 AM all 18.44 0.00 2.94 14.80 0.00
63.82
10:30:01 AM all 17.98 0.00 3.34 18.61 0.00
60.07
10:40:01 AM all 25.00 0.00 3.19 13.04 0.00
58.77
10:50:01 AM all 26.51 0.00 3.23 12.16 0.00
58.10
11:00:01 AM all 23.63 0.00 2.95 13.77 0.00
59.64
11:10:01 AM all 12.90 0.00 2.48 20.55 0.00
64.07
11:20:01 AM all 11.69 0.00 2.43 24.32 0.00
61.56
11:30:01 AM all 17.23 0.00 3.31 22.19 0.00
57.27
11:40:01 AM all 23.80 0.00 3.54 19.12 0.00
53.54
11:50:01 AM all 19.14 0.00 3.14 18.80 0.00
58.92
12:00:01 PM all 16.16 0.00 2.83 20.46 0.00
60.55
12:10:02 PM all 19.26 0.00 3.29 17.14 0.00
60.31
12:20:01 PM all 16.83 0.00 2.87 20.69 0.00
59.60
12:30:01 PM all 15.45 0.00 2.79 19.92 0.00
61.84
12:40:01 PM all 17.27 0.00 2.91 19.35 0.00
60.47
12:50:01 PM all 24.92 0.00 3.34 19.40 0.00
52.34
01:00:01 PM all 20.22 0.00 3.42 18.50 0.00
57.86
01:10:02 PM all 13.09 0.00 2.50 17.16 0.00
67.25
01:20:01 PM all 13.86 0.00 2.45 14.89 0.00
68.79
01:30:01 PM all 14.14 0.00 2.88 14.80 0.00
68.18
01:40:01 PM all 16.40 0.00 2.68 14.14 0.00
66.78
01:50:01 PM all 15.00 0.00 2.68 12.16 0.00
70.16
02:00:01 PM all 12.50 0.00 2.49 18.49 0.00
66.52
02:10:02 PM all 11.70 0.00 2.55 18.19 0.00
67.57
02:20:01 PM all 10.57 0.00 2.20 20.91 0.00
66.32
02:30:01 PM all 9.12 0.00 2.46 21.60 0.00
66.83
02:40:03 PM all 12.61 0.00 2.51 26.03 0.00
58.85
02:50:01 PM all 11.47 0.00 2.62 26.68 0.00
59.23
03:00:01 PM all 12.10 0.00 2.43 29.84 0.00
55.63
03:10:01 PM all 14.98 0.00 3.05 21.00 0.00
60.96
03:20:11 PM all 13.68 0.00 2.60 24.69 0.00
59.04
03:30:01 PM all 13.02 0.00 2.72 20.35 0.00
63.91
03:40:02 PM all 16.38 0.00 2.91 19.54 0.00
61.17
03:50:01 PM all 21.49 0.00 3.09 15.92 0.00
59.49
04:00:01 PM all 18.05 0.00 2.93 16.08 0.00
62.94
04:10:01 PM all 20.00 0.00 2.70 9.56 0.00
67.74
04:20:01 PM all 19.63 0.00 2.77 11.99 0.00
65.61
04:30:01 PM all 15.70 0.00 2.93 19.46 0.00
61.92
04:40:02 PM all 18.64 0.00 2.84 11.20 0.00
67.32
04:50:01 PM all 24.95 0.00 3.20 9.53 0.00
62.31
05:00:01 PM all 25.24 0.00 2.94 8.48 0.00
63.34
05:10:02 PM all 26.56 0.00 3.02 10.43 0.00
60.00
05:20:01 PM all 21.15 0.00 2.79 17.81 0.00
58.25
05:30:02 PM all 18.38 0.00 2.97 23.97 0.00
54.67
05:40:01 PM all 22.13 0.00 3.29 20.64 0.00
53.93
05:50:01 PM all 16.75 0.00 2.82 23.01 0.00
57.42
06:00:01 PM all 16.62 0.00 2.73 18.95 0.00
61.70
06:10:01 PM all 16.67 0.00 2.89 19.24 0.00
61.20
06:20:01 PM all 15.74 0.00 2.65 18.59 0.00
63.02
06:30:03 PM all 15.15 0.00 3.06 18.17 0.00
63.62
06:40:01 PM all 16.18 0.00 2.67 19.13 0.00
62.02
06:50:01 PM all 17.00 0.00 2.64 19.28 0.00
61.08
Average: all 17.45 0.01 2.90 17.01 0.00
62.62
Linux 2.6.18-371.1.2.el5PAE (edex2.atmos.albany.edu) 10/24/2013
avg-cpu: %user %nice %system %iowait %steal %idle
17.83 0.01 2.92 15.77 0.00 63.47
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 53.68 49.12 1906.29 4110764 159547826
sda1 0.00 0.02 0.00 1940 26
sda2 53.68 49.07 1906.29 4106962 159547800
sda3 0.00 0.02 0.00 1422 0
sdb 0.23 0.14 2.57 11415 215064
sdb1 0.00 0.02 0.02 1767 1408
sdb2 0.23 0.11 2.55 8912 213656
sdc 0.02 0.24 0.00 20098 32
sdc1 0.00 0.05 0.00 4184 16
sdc2 0.00 0.01 0.00 1045 0
sdc3 0.02 0.17 0.00 14133 16
sdd 147.31 569.81 4656.26 47690424 389707916
sdd1 147.31 569.79 4656.26 47688504 389707916
sdd2 0.00 0.01 0.00 1184 0
