7

I work with lxc user space tools on ubuntu 14.04 and I want to perform some stress tests and benchmarking within a container. I know that free and htop are not working properly in a container.

I am using dd and bonnie++ within a container to stress the hard disk it is a SSD.

Now on the host side, with iotop I can see the read and write io bandwidth used but in the cgroups I do have diverging results. The cgroups only capture a small part of the serviced bytes whereas iotop shows bandwidth usage of several hundred mega bytes.

In the cgroups, I'm capturing this entry: /sys/fs/cgroup/lxc/disk_stress/blkio.throttle.io_service_bytes

Any ideas why the values aren't equal? Which one are the correct ones?

Tropp Meaison
  • 71
  • 3

1 Answers1

9

The very, very bottom of the kernel documentation on blkio controller includes the note:

What works

  • Currently only sync IO queues are support. All the buffered writes are still system wide and not per group. Hence we will not see service differentiation between buffered writes between groups.

Practically, this means that write operations will appear in blkio.throttle.io_service_bytes only if they bypass kernel buffering.

The tool fio can illustrate this very easily. Direct, unbuffered writes should be reported in blkio.throttle.io_service_bytes:

fio --name wxyz --direct=1 --buffered=0 --size=1g --time_based --runtime=120s --bs=4k --rw=write --ioengine=sync --numjobs=1

Whereas with the opposite direct & buffered options, there is nothing reported in blkio.throttle.io_service_bytes, because writes pass through the kernel buffer cache and are scheduled later.

fio --name wxyz --direct=0 --buffered=1 --size=1g --time_based --runtime=120s --bs=4k --rw=write --ioengine=sync --numjobs=1

Additionally, this thread with a RedHat engineer who works on cgroups reiterates the point that once a write has passed to the write cache inside the kernel, "Due to this extra layer of cache, we lose the context information by the time IO reaches the device." And, so no accounting can occur by blkio.

stephen.z
  • 491
  • 4
  • 5
  • In the meantime, that Documentation link seems to have gotten broken. The current URL https://www.kernel.org/doc/Documentation/cgroup-v1/blkio-controller.txt doesn't seem to have this note about buffered writes...? – Josip Rodin Jan 26 '17 at 09:57