2

I have a problem with my VPS and disk I/O. My server is running nginx + PHP-FPM + APC. The database is located on another dedicated VPS. I have several WordPress MU sites living on the web server. The average I/O rate is 6k block/second.

I'm trying to understand what's causing the high I/O.

Output of 'free -m':

            total   used   free   shared   buffers   cached
Mem:         1005    973     31        0        96      568
-/+ buffers/cache:   307    697
Swap:         255      8    247

Output of 'iotop':

Total DISK READ: 0.00 B/s | Total DISK WRITE: 3.90 M/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
 2150 be/4 root        0.00 B/s    0.00 B/s  0.00 % 65.25 % [flush-202:0]
 6694 be/4 www-data    0.00 B/s   19.64 K/s  0.00 %  0.00 % php-fpm: pool www
 6700 be/4 www-data    0.00 B/s   23.56 K/s  0.00 %  0.00 % php-fpm: pool www
 8646 be/4 www-data    0.00 B/s  424.12 K/s  0.00 %  0.00 % php-fpm: pool www
10974 be/4 www-data    0.00 B/s   19.64 K/s  0.00 %  0.00 % php-fpm: pool www

The 'flush-202:0' process sometimes hits I/O of 99%. I've read this is the disk cache flushing process, but what causes it to run and how do I fix it?

Peter Mortensen
  • 2,319
  • 5
  • 23
  • 24
Mem
  • 301
  • 2
  • 4
  • 5
  • Ok, so what filesystem are you using? I think this is caused by using too much data in buffers, looks like [flush] is initiated by "sync" command. Try running "sync" manually - does it hit 99% of I/O? What hardware are you running on? Is this Hardware RAID card with it's own cache? Ahh.. VPS.. You probably don't know :( Let's do some search: http://linux.livejournal.com/1707762.html?style=mine Issue `vmstat -d`: what is it saying? –  Jan 22 '12 at 19:48
  • Possible duplicate of *[Limit Linux background flush (dirty pages)](https://serverfault.com/questions/126413)*. – Peter Mortensen Apr 21 '18 at 09:16
  • It sounds like your answer can be found in Server Fault question *https://serverfault.com/questions/126413/limit-linux-background-flush-dirty-pages*. – Zeki Feb 22 '11 at 01:03
  • Thanks a lot for the link. I'm going to try and experiment with changing the dirty_bytes value. Do you have any idea what process of the server could be causing this? – Mem Feb 22 '11 at 18:48
  • The flush process is just something that writes out memory pages to disk. This happens in the background even if you are not short on memory. The kernel will copy pages to disk so that if it does need more memory it can drop the pages from memory without having to copy them to disk at that time. It should only be doing this when the disk is otherwise idle. – Zeki Feb 23 '11 at 01:59

3 Answers3

2

I'm not sure that iotop sample shows something unusual. It's not a problem for the flush process to be a high percentage of your I/O at any point in time if there isn't much I/O going on at that time.

I would install atop, which can present real-time data like iotop, but has the advantage of also logging samples throughout the day. A day after installing it, I would open the logged data with atop -r log_filename, then go through the samples with t until I found times when the I/O reported in the system-level output is high. Then I would switch the per-process output to disk with d to see what processes were generating the I/O activity.

Peter Mortensen
  • 2,319
  • 5
  • 23
  • 24
sciurus
  • 12,493
  • 2
  • 30
  • 49
  • Atop's a great tool, but in this case, I think the OP has already found the info that atop would give them. – mc0e Aug 14 '13 at 12:55
  • I'm reading the atop documentation but I'm having a hard time understanding how it works with a log file. Could you describe it? I'm not seeing any log file by default. – felwithe Sep 09 '16 at 06:37
1

This may be a problem with APC.

If it works for you, in your PHP configuration, set:

apc.mmap_file_mask = /tmp/apc.shm.XXXXXX

If that doesn't work, remove the apc.mmap_file_mask setting altogether.

If not APC, then it's something else that uses disk backed caching. Like virtual memory for instance, or a varnish cache, or maybe something that uses DBM files. There are many possibilities there. Maybe even a database engine.

Users of APC should probably now migrate to Zend's opcache which comes with later free distributions of PHP (and earlier paid ones as I recall). Looking at the tuning of opcache might still be significant for managing your I/O load. https://www.sitepoint.com/understanding-opcache/ is a useful introduction, and links to several tools which can be used to check your cache performance (particularly the hit ratio).

Peter Mortensen
  • 2,319
  • 5
  • 23
  • 24
mc0e
  • 5,786
  • 17
  • 31
  • I don't have something called "apc.mmap_file_mask" in my php.ini-- in fact, the string "mmask" doesn't appear anywhere in /etc/php5 or subdirectories at all. – felwithe Sep 09 '16 at 05:46
0

You can achieve this with the pidstat program. Some distributions don't come with it installed. But you can download the sysstate package from here and compile it. Don't install it, but copy out the pidstat it compiles (or just run it in the current directory). You can pass the '-d' flag to get the output you want.

Peter Mortensen
  • 2,319
  • 5
  • 23
  • 24
Matthew Ife
  • 22,927
  • 2
  • 54
  • 71