Disk I/O and load average peaks once every hour

Question

We have updated our server from Debian Wheezy to Jessie and from php5.6 to php7.0, but now we have a disk I/O and load average peak exactly every hour. The exact time depends on the system start time.

On this server, we have:

nginx/1.10.1
PHP 7.0.8-1~dotdeb+8.1
Percona mysql server 5.6.30-76.3-log
dovecot 2.2.devel
postfix 2.11.3-1
java 1.7.0_101

We have tried to return to php5.6, disable cron, disable postfix and dovecot, stop our Java app, but nothing helped.

The peaks look like the following:

The iotop looks like the following:

How can I know exactly what causes these peaks and eliminate them?

score 3 · Answer 1 · answered Jun 28 '16 at 13:48

3

The problem was https://developers.google.com/speed/pagespeed/module/ which have done cache clean every hour.

Configuration it to store files on tmpfs fixed the problem.

answered Jun 28 '16 at 13:48

dima_mak

125
1
7

score 1 · Answer 2 · edited Apr 13 '17 at 12:13

1

Scientific Method is your friend.

You or someone on your behalf will have to monitor your system to gather the necessary data to allow you to make an informed decision.

Your load average spikes quite sharply and at predictable times, that means it should be fairly straightforward to monitor the event with tools like top, iostat, vmstat etc and start the journey.

edited Apr 13 '17 at 12:13

Community

1

answered Jun 28 '16 at 08:31

user9517

114,104
20
206
289

That's what we try for 4 days, but without results... – dima_mak Jun 28 '16 at 09:08
You will have to just have to roll up your sleeves and learn to gather the right data. We can't help you via SF as we need the data you don't have. This is a Q&A site not a remote diagnostics/tech support site. – user9517 Jun 28 '16 at 09:12
I understand, but i hope may be some one will suggest what we need to check more. And I also can provide the needed data – dima_mak Jun 28 '16 at 10:09

score 0 · Accepted Answer · answered Jun 28 '16 at 08:41

Maybe you have some MySQL scheduled events going on each hour?

MySQL Events are tasks that run according to a schedule. Therefore, we sometimes refer to them as scheduled events. When you create an event, you are creating a named database object containing one or more SQL statements to be executed at one or more regular intervals, beginning and ending at a specific date and time. Conceptually, this is similar to the idea of the Unix crontab (also known as a “cron job”) or the Windows Task Scheduler.

There is no events in information_schema.EVENTS – dima_mak Jun 28 '16 at 09:07 — dima_mak, Jun 28 '16 at 09:07

Disk I/O and load average peaks once every hour

3 Answers3