Re: Rather large LA

vmstat 1 and iostat -x output

Normal

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
3 0 2332 442428 73904 31287344 0 0 89 42 0 0 7 5 85 3 0
4 1 2332 428428 73904 31288288 0 0 1440 0 6553 29066 5 2 91 1 0
4 1 2332 422688 73904 31288688 0 0 856 0 4480 18860 3 1 95 1 0
0 0 2332 476072 73920 31289444 0 0 544 1452 4478 19103 3 1 95 0 0
3 0 2332 422288 73920 31290572 0 0 1268 496 5565 23410 5 3 91 1 0

cavg-cpu: %user %nice %system %iowait %steal %idle
5.11 0.01 2.58 2.56 0.00 89.74

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 1.00 143.00 523.50 108.00 8364.00 2008.00 16.42 2.78 4.41 1.56 98.35
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda2 1.00 143.00 523.50 108.00 8364.00 2008.00 16.42 2.78 4.41 1.56 98.35
sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %system %iowait %steal %idle
4.89 0.00 2.94 3.14 0.00 89.04

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 1.00 0.00 285.00 0.00 4808.00 0.00 16.87 2.46 8.29 3.02 86.20
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda2 1.00 0.00 285.00 0.00 4808.00 0.00 16.87 2.46 8.29 3.02 86.20
sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 161.50 0.00 6.50 0.00 1344.00 206.77 0.00 0.69 0.15 0.10
sdb1 0.00 161.50 0.00 6.50 0.00 1344.00 206.77 0.00 0.69 0.15 0.10

After Restart

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 34 2332 5819012 75632 25855368 0 0 89 42 0 0 7 5 85 3 0
4 39 2332 5813344 75628 25833588 0 0 5104 324 5480 27047 3 1 84 11 0
2 47 2332 5815212 75336 25812064 0 0 4356 1664 5627 28695 3 1 84 12 0
2 40 2332 5852452 75340 25817496 0 0 5632 828 5817 28832 3 1 84 11 0
1 45 2332 5835704 75348 25817072 0 0 4960 1004 5111 25782 2 1 88 9 0
2 42 2332 5840320 75356 25811632 0 0 3884 492 5405 27858 3 1 88 8 0
0 47 2332 5826648 75348 25805296 0 0 4432 1268 5888 29556 3 1 83 13 0

avg-cpu: %user %nice %system %iowait %steal %idle
3.26 0.00 1.69 25.21 0.00 69.84

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.50 45.00 520.00 2.50 8316.00 380.00 16.64 71.70 118.28 1.92 100.10
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda2 0.50 45.00 520.00 2.50 8316.00 380.00 16.64 71.70 118.28 1.92 100.10
sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 196.50 0.00 10.50 0.00 1656.00 157.71 0.01 0.67 0.52 0.55
sdb1 0.00 196.50 0.00 10.50 0.00 1656.00 157.71 0.01 0.67 0.52 0.55

avg-cpu: %user %nice %system %iowait %steal %idle
3.97 0.00 1.71 20.88 0.00 73.44

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 1.00 0.00 532.00 0.00 8568.00 0.00 16.11 73.73 148.44 1.88 100.05
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda2 1.00 0.00 532.00 0.00 8568.00 0.00 16.11 73.73 148.44 1.88 100.05
sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 106.50 0.00 11.50 0.00 944.00 82.09 0.00 0.39 0.30 0.35
sdb1 0.00 106.50 0.00 11.50 0.00 944.00 82.09 0.00 0.39 0.30 0.35

Regards

Richard

.........

On 5 Sep 2011, at 21:05, Alan Hodgson wrote:

> On September 5, 2011, Richard Shaw <richard(at)aggress(dot)net> wrote:
> > Hi Andy,
> >
> > It's not a new issue no, It's a legacy system that is in no way ideal but
> > is also not in a position to be overhauled. Indexes are correct, tables
> > are up to 25 million rows.
> >
> > On startup, it hits CPU more than IO, I'll provide some additional stats
> > after I restart it tonight.
>
> I bet it's I/O bound until a good chunk of the active data gets cached. Run a vmstat 1 while it's that busy, I bet most of the cpu time is really in io_wait.