14

I have a cron that basically do a simple "tar zcf" in the night.

The server has:

  • 8 Cores - Intel(R) Xeon(R) CPU E5606 @ 2.13GHz
  • 25GB RAM
  • Ubuntu 12.04.2 LTS
  • Hardware RAID 1 (LSI Logic / Symbios Logic MegaRAID SAS SMC2108) with two 2.728TB harddrives

As you can see on the monitoring screenhost:

http://clip2net.com/s/57YRKP

During almost all the time of the tar, disc I/O goes to >90% and make all other app (mysql, apache) to slow down a lot.

2 questions:

  • Is it normal to have so high disc I/O during the backup?
  • Is there a way to limit the disc I/O so other app can continue to work correctly?

Thank you!

acemtp
  • 253
  • 2
  • 6

4 Answers4

13

It's expected to see high I/O during backups because they're generally made over large file trees with large files. You can use ionice to prioritize I/O jobs in Linux with classes and levels. IIRC, class 2, level 7 is the lowest, non starving level which will make it practically invisible to other I/O loads and users. See man ionice for usage and details.

bayindirh
  • 624
  • 1
  • 5
  • 14
11

Beside the rather general approach with ionice there is a nice device mapper target (ioband) which allows precise control over the bandwidth to a (DM) block device. Unfortunately it is not part of the standard kernel.

Furthermore you can probably speed up tar by

  1. Reading the file names into the disk cache: find /source/path -printf ""
  2. Reading the inodes into the disk cache: find /source/path -perm 777 -printf ""
  3. Making tar read and write larger blocks from and to the disk by e.g. using a pipe with mbuffer or buffer (with at least 100 MiB of RAM): tar ... | mbuffer -m 256M -P 100 -p 1 ...
Hauke Laging
  • 5,157
  • 2
  • 23
  • 40
  • Why does reading the file names/inodes into the cache reduce disk IO while tar'ing? I would expect it to increase the average IO while reducing the total time only slightly. – scai May 27 '13 at 11:13
  • 3
    @scai This does not help with SSDs; my recommendation refers to spinning harddisks only. What kills performance with those is head movement. The file names are stored in continuous blocks, the inodes are stored in continuous blocks and the file content is stored in continuous blocks. If you do it the tar way then you read the file (and subdirectory) names of one directory, access the inode for one file, then the file itself, then the inode for the next file, then the next file itself... That causes more head movement than reading all names and inodes after each other. – Hauke Laging May 27 '13 at 11:15
  • @scai The performance impact depends on what you do. It is rather small for full backups (probably depends on the file sizes) but I noticed a big difference for differential backups (not for tar, though, as I don't use that but this should be a general effect). – Hauke Laging May 27 '13 at 11:18
  • Just to be sure I understood correctly. For 1. and 2., we just have to call the find command and Linux will automatically cache it? – acemtp May 27 '13 at 12:38
  • @acemtp That is correct. `find` without (e.g.) `-perm` will not access the file inode, though. But that allows for the optimization to use two `find` calls. If you make the same `find` call twice (with little time in between), the second one is usually going to finish within seconds (or less). Depending on the amount of free memory and the amount of data cached at a certain point the data is thrown out of the cache. Reading too much may thus just slow down the operation. If you can feed the backup program with file names via stdin then you can prevent this by reading blocks of e.g. 100 files. – Hauke Laging May 27 '13 at 13:05
  • doesn't give great caching results: http://pastie.org/7969279 – acemtp May 27 '13 at 15:31
  • @acemtp That is strange. If I do the same here then I get "real 0m59.017s" vs. "real 0m0.806s": a 74 times performance increase. Considering my rather slow disk and the multiple amount of time this takes on your system the problem may be that it is enough file system objects to cause cache trashing. If you replace `-printf ""` by `-printf .` and add ` | wc` then you get the number of objects found. That is 181192 in my case. You can limit the number to 100,000 by replacing `wc` with `| dd if=/dev/stdin of=/dev/null bs=1000 count=100`. Compare that. – Hauke Laging May 27 '13 at 15:59
  • @HaukeLaging There's really something not normal. "find /home/ -printf . |wc" gave me 0 1 137660 so 137 660 files. Not so much comparing to your test. And It takes 8mn where it takes < 1mn? I tried a "hdparm -Tt /dev/sdb" and I have Timing cached reads: "11478 MB in 2.00 seconds = 5744.90 MB/sec" and "Timing buffered disk reads: 62 MB in 4.74 seconds = 13.09 MB/sec" – acemtp May 27 '13 at 16:16
  • @HaukeLaging I did more test and there's something definitively strange, I posted another question since it's not related to this question: http://serverfault.com/questions/511356/why-parsing-directories-are-so-slow – acemtp May 28 '13 at 11:53
1

I would recommend ditching tar and going with rsync (as mentioned by Dogsbody). I use BackupPC to backup files on my Windows and Linux systems and it supports using tar as well as rsync and automatically takes care of the hard linking for you as well as provides a nice web interface.

http://backuppc.sourceforge.net/

Atari911
  • 375
  • 1
  • 7
0

As others have replied, yes this is normal, and ionice is a good generic way of not letting it affect your system.

A number of times I've seen people tar things up when they don't need to though. If any percentage of the data you are copying hasn't changed since the last copy then I'd suggest giving rsync a try.

This will reduce IO by only copying the files that have changed since last copy. you won't be able to reduce the IO by more than half as all the data would still need to be read but you will significantly reduce the amount of data written (which depending on your hardware can be a slower operation as well).

If you want separate copies/backups each time it's run then the most powerful option is –link-dest which allows you to hard link unchanged files to a previous backup. This saves HUGE amounts of space on the backup server. e.g. I backup a machine (Fred), Fred has a 20GB HD and I back up/copy the entire drive excluding /proc and /dev. I now have a 20GB directory on my backup server. The next day I backup Fred again and –link-dest to yesterdays backup. Rsync compares the remote files with the local copy and if exactly the same will not bother transferring them but will hard link the new file to yesterdays file. Any files that have changed are copied down a fresh (or partially copied using yesterdays backup if possible). If only 100MB of files of changed since yesterday I now have two directories both with 20GB of files but only taking up 20.1Gb of space on the backup server!

I hope that helps and still answers your question.

Dogsbody
  • 656
  • 4
  • 15