If I run iostat I saw this:
avg-cpu: %user %nice %system %iowait %steal %idle
15.36 0.78 9.66 21.46 0.00 52.74
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 180.34 494.08 5741.44 28772164 334347968
sdc 0.01 0.08 0.00 4492 8
sdb 0.01 0.09 0.01 5412 736
sdd 0.32 6.64 12.68 386732 738656
dm-0 93.32 110.10 731.99 6411692 42626904
dm-1 0.01 0.05 0.00 2680 0
dm-2 647.62 383.52 5009.47 22333866 291722528
Looks like dm-0, dm-2 is the culprit. Most likely it's read turning into write because iotop shows:
10583 be/4 nobody 0.00 B/s 7.54 K/s 0.00 % 0.00 % httpd -k start -DSSL
10582 be/4 nobody 0.00 B/s 7.54 K/s 0.00 % 0.00 % httpd -k start -DSSL
10705 be/4 nobody 0.00 B/s 7.54 K/s 0.00 % 0.00 % httpd -k start -DSSL
10733 be/4 nobody 0.00 B/s 7.54 K/s 0.00 % 0.00 % httpd -k start -DSSL
10766 be/4 nobody 0.00 B/s 7.54 K/s 0.00 % 0.00 % httpd -k start -DSSL
8894 be/4 nobody 0.00 B/s 7.54 K/s 0.00 % 0.00 % httpd -k start -DSSL
10974 be/4 nobody 0.00 B/s 7.54 K/s 0.00 % 0.00 % httpd -k start -DSSL
9059 be/4 nobody 0.00 B/s 7.54 K/s 0.00 % 0.00 % httpd -k start -DSSL
9156 be/4 nobody 0.00 B/s 7.54 K/s 0.00 % 0.00 % httpd -k start -DSSL
1011 be/4 nobody 0.00 B/s 7.54 K/s 0.00 % 0.00 % httpd -k start -DSSL
7272 be/4 nobody 0.00 B/s 7.54 K/s 0.00 % 0.00 % httpd -k start -DSSL
11409 be/4 nobody 0.00 B/s 7.54 K/s 0.00 % 0.00 % httpd -k start -DSSL
11445 be/4 nobody 0.00 B/s 7.54 K/s 0.00 % 0.00 % httpd -k start -DSSL
11447 be/4 mortgage 0.00 B/s 60.34 K/s 0.00 % 0.00 % [php]
4298 be/4 nobody 0.00 B/s 7.54 K/s 0.00 % 0.00 % httpd -k start -DSSL
11454 be/4 mortgage 0.00 B/s 218.73 K/s 0.00 % 0.00 % [php]
5698 be/4 nobody 0.00 B/s 7.54 K/s 0.00 % 0.00 % httpd -k start -DSSL
1440 be/4 nobody 0.00 B/s 7.54 K/s 0.00 % 0.00 % httpd -k start -DSSL
10030 be/4 nobody 0.00 B/s 7.54 K/s 0.00 % 0.00 % httpd -k start -DSSL
10061 be/4 nobody 0.00 B/s 7.54 K/s 0.00 % 0.00 % httpd -k start -DSSL
10093 be/4 nobody 0.00 B/s 7.54 K/s 0.00 % 0.00 % httpd -k start -DSSL
10094 be/4 nobody 0.00 B/s 7.54 K/s 0.00 % 0.00 % httpd -k start -DSSL
6073 be/4 nobody 0.00 B/s 7.54 K/s 0.00 % 0.00 % httpd -k start -DSSL
10231 be/4 nobody 0.00 B/s 7.54 K/s 0.00 % 0.00 % httpd -k start -DSSL
2048 be/4 named 0.00 B/s 0.00 B/s 0.00 % 0.00 % named -u named
1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % init
2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd]
3 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/0]
4 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0]
5 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/0]
6 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/0]
My fstab is this:
/dev/mapper/VolGroup-lv_root / ext4 usrjquota=quota.user,jqfmt=vfsv0 1 1
UUID=1450c2bf-d431-4621-9e8e-b0be57fd79b6 /boot ext4 defaults 1 2
/dev/mapper/VolGroup-lv_home /home ext4 usrjquota=quota.user,jqfmt=vfsv0 1 2
/dev/mapper/VolGroup-lv_swap swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/usr/tmpDSK /tmp ext3 defaults,noauto 0 0
/dev/sdb1 /home2 auto auto,defaults 0 0
/dev/sdc1 /home3 auto auto,defaults 0 0
/dev/sdd1 /home4 auto auto,defaults 0 0
The problem is which one is dm-0, dm1, and dm-2? I suppose I'll just change defaults to defaults,relatime, but which rows?
if I do ls -l /dev/disk/by-id/
I got
lrwxrwxrwx. 1 root root 10 Dec 26 05:03 dm-name-VolGroup-lv_home -> ../../dm-2
lrwxrwxrwx. 1 root root 10 Dec 26 05:03 dm-name-VolGroup-lv_root -> ../../dm-0
lrwxrwxrwx. 1 root root 10 Dec 26 05:03 dm-name-VolGroup-lv_swap -> ../../dm-1
lrwxrwxrwx. 1 root root 10 Dec 26 05:03 dm-uuid-LVM-U0gpPI6UFcotbgHEwlPqKQOw6tPYQ1pjamluUcU2YJ5xYPp2re51sVkE0LyxI2qy -> ../../dm-2
lrwxrwxrwx. 1 root root 10 Dec 26 05:03 dm-uuid-LVM-U0gpPI6UFcotbgHEwlPqKQOw6tPYQ1pjjec2s02YQq8Dpw65ynU42qYnSWuTlPc2 -> ../../dm-0
lrwxrwxrwx. 1 root root 10 Dec 26 05:03 dm-uuid-LVM-U0gpPI6UFcotbgHEwlPqKQOw6tPYQ1pjVufBZh0L0hG8uFJ9cWxsBkEJQ0OvNX1H -> ../../dm-1
There are 2 entries for /dm-0 to dm-2. I wonder why. I suppose I got what I need for now. As I suspect, the swap is not the issue. Root and home is the issue which is kind of strange given that all accounts at /home has already been moved somewhere else.
