7

My tar command 

tar --remove-files -cvif 2011-08-02_14-05-09.tar 2011-08-02_14-05-09

Writes 80MB/s. That's beyond the capacity we can spare because MySQL starts queuing up writes and eventually we get monitoring alerts cause systems won't wait forever on MySQL and time out.

Q: Can I throttle tar to only do 20MB/s?

Didn't find anything in the manpage and we already tried with the lowest ionice class: /usr/bin/ionice -c3 tar <...> but that doesn't impact the MBs written/s and MySQL still freezes up.

Maybe ionice is ignored because we're on a /dev/md0 software RAID device?

kvz
  • 402
  • 4
  • 14
  • I guess as a hack we could immediately let tar bzip the archive cause we do have CPU cycles to spare. But I'm really interested to see if I can cap IO in a solid manner. – kvz Aug 02 '11 at 14:39
  • `ionice` not working for you may have been because you were running with the incorrect scheduler. It apparently only works with the CFQ scheduler. – Stu Thompson Oct 31 '11 at 12:28

2 Answers2

5

You can use ratepipe.

I downloaded and tried it out and it seems to DTRT. It should work for you:

tar --remove-files -cvif - 2011-08-02_14-05-09 | ratepipe -r 20 > 2011-08-02_14-05-09.tar
MikeyB
  • 38,725
  • 10
  • 102
  • 186
  • 2
    Great idea! I found an alternative called `pv` (http://linux.die.net/man/1/pv) that can be found in apt. Syntax would become `tar --remove-files -cvif - 2011-08-02_14-05-09 | pv -L 20m > 2011-08-02_14-05-09.tar` Can also e.g. show a progress bar – kvz Aug 02 '11 at 15:32
  • 1
    That's it! I knew there was a sexier program around, but couldn't remember `pv`. – MikeyB Aug 02 '11 at 16:09
  • This is a gem of a find. Thanks. `ratepipe` rocks. – Stu Thompson Oct 31 '11 at 10:37
1

You could probably write the tar to stdout (using -f -) and pipe it into something that throttles. I'll see if I can find something quickly.

Since tar will not buffer much, it should wait on the reads.

Edit: MikeB beat me to it: ratepipe will do.

Joris
  • 5,939
  • 1
  • 15
  • 13