Redis rdb persistence quite slow

2

My two nodes in a Redis cluster have quite different persistence performance. The master node is always much faster than the slave one when saving data to disk. Here is the persistence info for master:

[fred@redis_master temp]$ redis-cli -h 192.168.1.151 -p 8382
192.168.1.151:8382> info persistence
# Persistence
loading:0
rdb_changes_since_last_save:1206
rdb_bgsave_in_progress:0
rdb_last_save_time:1529636130
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:28
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
192.168.1.151:8382>

And below is the info for slave:

[fred@redis_slave temp]$ redis-cli -h 192.168.2.151 -p 8381
192.168.2.151:8381> info persistence
# Persistence
loading:0
rdb_changes_since_last_save:4850
rdb_bgsave_in_progress:0
rdb_last_save_time:1529635749
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:50
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
192.168.2.151:8381>

It's can be seen that the master spent 28s on "rdb_last_bgsave_time_sec" while the slave spent 50s, which has been truely slowing down the slave node. I suspect it is most likely something to do with the hard-drive performance so I did the following tests:

[fred@redis_master temp]$ time dd if=/dev/zero of=test.dbf bs=8k count=5000 oflag=direct
5000+0 records in
5000+0 records out
40960000 bytes (41 MB) copied, 0.329285 s, 124 MB/s

real    0m0.331s
user    0m0.006s
sys 0m0.119s

[fred@redis_slave temp]$ time dd if=/dev/zero of=test.dbf bs=8k count=5000 oflag=direct
5000+0 records in
5000+0 records out
40960000 bytes (41 MB) copied, 88.126 s, 465 kB/s

real    1m28.127s
user    0m0.008s
sys 0m0.210s

Apparently the slave spent dramatically more time than the master to produce just a 41MB file. Can I simply conclude that the slow persistence on the slave node is completely due to the bad hard-drive? Is there any other more effective way to debug this persistence problem?

I will appreciate it if someone can help.

Andrea Lau

Posted 2018-06-22T03:51:32.447

Reputation: 21

No answers