3

Background:

I use a cluster with my coworkers and if anyone writes too many big files at the same time I can't even move the cursor in vim (the horror!). I'm not sure if it is more related to the number of files open or the "bandwith" of hard drive being used but it definitely only happens when a big write operation is taking place (including downloading big files). I know you can limit hard drive space by quotas per user, but I've never heard of being able to limit data transfer speeds.

Question

Is it possible to limit the amount of data written per second by a user in linux?

sequenceGeek
  • 155
  • 1
  • 6
  • 5
    gotta run, but look into `ionice(1)` – sarnold Jan 19 '12 at 00:36
  • perhaps something like PAM, ulimit? http://www.linuxtopia.org/online_books/linux_administrators_security_guide/16_Linux_Limiting_and_Monitoring_Users.html#Limiting_users –  Jan 19 '12 at 00:47
  • For ionice info: http://friedcpu.wordpress.com/2007/07/17/why-arent-you-using-ionice-yet/ –  Jan 19 '12 at 00:49
  • @sequenceGeek: you might want to look at the hard drive's SMART data and see if it has been having trouble. – wallyk Jan 19 '12 at 00:58
  • @wallyk: thanks for the tip - I didn't know about SMART up until now. I think it solely has to do with the write speed, however. Networked cluster with NFS = problems :( –  Jan 19 '12 at 02:29
  • Write speed should not blank out your SSH connection. its likely caused by a saturated network interface or similar. try limiting the protocol they use to upload. QoS can also help (but is more complicated to setup) – Silverfire Jan 19 '12 at 03:50
  • @Silverfire: It seems like I can throttle network bandwith for SSH packets with QoS, but everything we do is over ssh... So if my co-worker is running a python script that is writing a lot of data through a ssh connection is there a way to hone in on just such processes? – sequenceGeek Jan 20 '12 at 02:04

3 Answers3

3

Use cgroups together with CFQ I/O scheduler (the default for many distributions). CFQ is aware of cgroups and can give any user, group or process only x% of disk I/O time.

So, if you have one cgroup called sequenceGeek, having 90% of maximum resources, you can then have another cgroup called coworkers, having 10% of resources. Or something similarly fair.

man cgrules.conf, man cgset and man cgconfig.conf should get you started. This at least in Fedora 16, did not have time to check other distributions right now if they have similar config file names. Probably they do.

EDIT: Oh, just noticed from your comments you are using NFS. Then your network might be saturated and you need to use QoS with tc and iptables. Google for Advances Linux Routing How-To, it has ready-made examples of throttling the traffic and creating priority classes. Or if possible, do the throttling at your firewall/router/load balancer.

Could also be an old kernel version having issues with NFS or bad NFS mount options.

Janne Pikkarainen
  • 31,454
  • 4
  • 56
  • 78
1

If you cannot move your cursor, my guess is that your CPU is waiting on IO for a very high amount of time. You can check this by running top, and looking at the amount listed at %wa near the top.

The blkio controller of the aforementioned cgroups is a method of controlling the amount of IO someone or some group can do. I'm in a bit of a hurry right now, but if you read the kernel docs (/usr/share/doc/kernel-<version>/Documentation/cgroups), I'm sure you'll manage. You will need a fairly recent kernel though, but I think anything above 2.6.32 (the EL6 kernel) will probably do.

You need to configure the blkio.throttle.write_bps_device file in your cgroups filesystem. With that, together with the /etc/cgconfig.conf and /etc/cgrules.conf, you can limit any user or group in the amount of bandwidth on your storage.

wzzrd
  • 10,269
  • 2
  • 32
  • 47
0

I'm not entirely sure, but after reading a bit from man limits.conf, and assuming you're root, I'd suggest to add hard limits for other people's accounts on data (maximum data size), priority and nice. Something like setting their limits lower than yours, and then setting whichever app you're running's priority to higher than they can. Of course, you should research a bit more and test before making any changes, as it might seriously interfere with their user experience.

Good luck.

Misguided
  • 101