How to run docker build at lower priority?

Question

I often have to re-build Docker images, and I do that inside the host I run my containers in. Occasionally I see that this puts considerable pressure on the CPU, so I thought I could run docker build under nice -n19, but this seems to make no difference in terms of how much docker yields when other processes need to run.

I'd rather not use Docker Hub repos, because this is private stuff and I'm trying to save every penny I can right now. I also know that I could set up another machine as a build/repo – I could use a machine at the office, for example – but I'm not sure how.

So, question: why does nice -n19 docker build ... seem to not be very helpful?

(bonus points for pointing me at docs on how to set up my own private build/repo machine)

Lie Ryan · Accepted Answer · 2016-04-28T13:45:53.593

why does nice -n19 docker build ... seem to not be very helpful?

The docker commands is a client of the docker daemon, which spawns the processes that runs in the containers. When you give a lower priority to the docker command, the priority of the docker daemon itself is unaffected, thus your build and exec runs at their default priority. It's like giving lower priority to your web browser does not mean that the web server is going to service your requests at lower priority.

how to set up my own private build/repo machine

For a basic separate build machine, you can just do docker export, scp, docker import. But for a more serious build system though, you may want to run a private docker registry. Some more useful documentations:

If you run your own private registry, you can do a build from your local workstation in office, then use docker push and docker pull to upload the docker image to your private registry and fetch it to wherever it needs to be.

score 1 · Answer 2 · answered Jan 28 '21 at 22:14

Instead of using nice, consider using the other options the CPU management docker API gives you:

docker build --help | grep cpu
      --cpu-period int          Limit the CPU CFS (Completely Fair Scheduler) period
      --cpu-quota int           Limit the CPU CFS (Completely Fair Scheduler) quota
  -c, --cpu-shares int          CPU shares (relative weight)
      --cpuset-cpus string      CPUs in which to allow execution (0-3, 0,1)
      --cpuset-mems string      MEMs in which to allow execution (0-3, 0,1)

For example, I've recently updated some of my builds to reserve 1 of 6 of the CPU cores I have on one of our build machines with a command like this:

docker build --cpuset-cpus 0-4 .

It doesn't appear that any of these options are backed by a nice value so you might want to experiment to see what works best for you. I posted some quick tests I ran but these could be improved by running a longer test and having something using CPU on the host to better simulate sharing.

--cpu-period="100000" --cpu-quota="150000"

docker run --name 7z --rm -it --cpu-period="100000" --cpu-quota="150000" localhost/7z bash -c 'time dd if=/dev/zero bs=50K count=1K | 7zr -si a out.7z'

7-Zip (a) [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=C,Utf16=off,HugeFiles=on,64 bits,12 CPUs Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz (306F2),ASM,AES-NI)

Creating archive: out.7z

Items to compress: 1

  0M1024+0 records in
1024+0 records out
52428800 bytes (52 MB, 50 MiB) copied, 0.0942925 s, 556 MB/s
                  
Files read from disk: 1
Archive size: 7815 bytes (8 KiB)
Everything is Ok

real    0m2.326s
user    0m3.078s
sys     0m0.359s

--cpu-shares 500

docker run --name 7z --rm -it --cpu-shares 500 localhost/7z bash -c 'time dd if=/dev/zero bs=50K count=1K | 7zr -si a out.7z'

7-Zip (a) [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=C,Utf16=off,HugeFiles=on,64 bits,12 CPUs Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz (306F2),ASM,AES-NI)

Creating archive: out.7z

Items to compress: 1

  0M1024+0 records in
1024+0 records out
52428800 bytes (52 MB, 50 MiB) copied, 0.08448 s, 621 MB/s
                  
Files read from disk: 1
Archive size: 7815 bytes (8 KiB)
Everything is Ok

real    0m1.982s
user    0m2.961s
sys     0m0.343s

nice

docker run --name 7z --rm -it localhost/7z bash -c 'time dd if=/dev/zero bs=50K 
count=1K | nice 7zr -si a out.7z'

7-Zip (a) [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=C,Utf16=off,HugeFiles=on,64 bits,12 CPUs Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz (306F2),ASM,AES-NI)

Creating archive: out.7z

Items to compress: 1

  0M1024+0 records in
1024+0 records out
52428800 bytes (52 MB, 50 MiB) copied, 0.0928115 s, 565 MB/s
                  
Files read from disk: 1
Archive size: 7815 bytes (8 KiB)
Everything is Ok

real    0m2.026s
user    0m3.066s
sys     0m0.375s

--cpuset-cpus 0-1

docker run --name 7z --rm -it --cpuset-cpus 0-1 localhost/7z bash -c 'time dd if=/dev/zero bs=50K count=1K | 7zr -si a out.7z'

7-Zip (a) [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=C,Utf16=off,HugeFiles=on,64 bits,12 CPUs Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz (306F2),ASM,AES-NI)

Creating archive: out.7z

Items to compress: 1

  0M1024+0 records in
1024+0 records out
52428800 bytes (52 MB, 50 MiB) copied, 0.0868617 s, 604 MB/s
                  
Files read from disk: 1
Archive size: 7815 bytes (8 KiB)
Everything is Ok

real    0m1.908s
user    0m2.432s
sys     0m0.295s

nothing (control)

docker run --name 7z --rm -it localhost/7z bash -c 'time dd if=/dev/zero bs=50K count=1K | 7zr -si a out.7z'                                                                                               1 ↵

7-Zip (a) [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=C,Utf16=off,HugeFiles=on,64 bits,12 CPUs Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz (306F2),ASM,AES-NI)

Creating archive: out.7z

Items to compress: 1

  0M1024+0 records in
1024+0 records out
52428800 bytes (52 MB, 50 MiB) copied, 0.0938064 s, 559 MB/s
                  
Files read from disk: 1
Archive size: 7815 bytes (8 KiB)
Everything is Ok

real    0m2.036s
user    0m3.021s
sys.    0m0.389s

score 0 · Answer 3 · answered Oct 30 '19 at 18:20

Lie's answer gives the reason Docker won't nice your builds. I wanted to give you a practical workaround, below.

One way you can make your Docker builds more gentle is to identify the resource intensive parts of your Dockerfile, and then just nice those parts in the Dockerfile itself. This will also make your build more gentle on remote build infrastructure (example Jenkins CI).

In my case, I was building a pretty heavy Apache mod_perl container with a ton of CPAN modules. Building all those modules was the expensive part, since it compiles a lot of code using GCC. Anyhow, I was able to nice this step of my build by changing the line of my Dockerfile from

RUN cpanm --quiet --installdeps --local-lib-contained /perlmod .

to

RUN nice -n19 cpanm --quiet --installdeps --local-lib-contained /perlmod .

Even though this is janky and a bit hacky, I'm not sure I would publish sofware with nice in the Dockerfile, but you said this was for private stuff anyway, so it's probably a fitting solution.

You could generalize this to any other expensive operation in a Dockerfile,

Other Examples:

Python

RUN nice -n19 pip install some_module

Node.js

RUN nice -n19 npm install .

An expensive compression step

RUN nice -n19 xz some_file.bin