cannot start docker when using chroot

Question

I am trying to build a custom distribution that runs docker. I am using kiwi framework to achieve that. Docker installation is done successfully during the build. The problem is that I want to load some docker images during the build process so that the system includes all needed images when it starts. The problem is that the KIWI software uses chroot to run commands like docker load, docker pull. These commands need a running docker daemon. When I try to start the deamon, it always fails. I tried the following things:

running docker through : systemctl start docker failed (systemctl by design will refuse starting a service in chroot jail)

running docker through: /usr/bin/dockerd also fails and generates the below logs

time="2019-02-04T16:00:25.861184013+01:00" level=warning msg="Error while setting daemon root propagation, this is not generally critical but may cause some functionality to not work or fallback to less desirable behavior" dir=/var/lib/docker error="error getting daemon root's parent mount: Can't find mount point of /var/lib/docker"
time="2019-02-04T16:00:25.861693151+01:00" level=info msg="libcontainerd: started new docker-containerd process" pid=19256
time="2019-02-04T16:00:25.861719642+01:00" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2019-02-04T16:00:25.861728310+01:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2019-02-04T16:00:25.861756358+01:00" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///var/run/docker/containerd/docker-containerd.sock 0  <nil>}]" module=grpc
time="2019-02-04T16:00:25.861765096+01:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2019-02-04T16:00:25.861797149+01:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4203c57b0, CONNECTING" module=grpc
time="2019-02-04T16:00:25+01:00" level=info msg="starting containerd" revision=468a545b9edcd5932818eb9de8e72413e616e86e version=v1.1.2 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.content.v1.content"..." type=io.containerd.content.v1 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." type=io.containerd.snapshotter.v1 
time="2019-02-04T16:00:25+01:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="failed to find the mount info for "/var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs"" 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.aufs"..." type=io.containerd.snapshotter.v1 
time="2019-02-04T16:00:25+01:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.aufs" error="modprobe aufs failed: "modprobe: FATAL: Module aufs not found in directory /lib/modules/4.12.14-lp150.12.45-default\n": exit status 1" 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.native"..." type=io.containerd.snapshotter.v1 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." type=io.containerd.snapshotter.v1 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1 
time="2019-02-04T16:00:25+01:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.zfs" error="failed to find the mount info for "/var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs"" 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." type=io.containerd.metadata.v1 
time="2019-02-04T16:00:25+01:00" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="failed to find the mount info for "/var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs"" 
time="2019-02-04T16:00:25+01:00" level=warning msg="could not use snapshotter aufs in metadata plugin" error="modprobe aufs failed: "modprobe: FATAL: Module aufs not found in directory /lib/modules/4.12.14-lp150.12.45-default\n": exit status 1" 
time="2019-02-04T16:00:25+01:00" level=warning msg="could not use snapshotter zfs in metadata plugin" error="failed to find the mount info for "/var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs"" 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." type=io.containerd.differ.v1 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." type=io.containerd.gc.v1 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.service.v1.containers-service"..." type=io.containerd.service.v1 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.service.v1.content-service"..." type=io.containerd.service.v1 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.service.v1.diff-service"..." type=io.containerd.service.v1 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.service.v1.images-service"..." type=io.containerd.service.v1 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.service.v1.leases-service"..." type=io.containerd.service.v1 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.service.v1.namespaces-service"..." type=io.containerd.service.v1 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.service.v1.snapshots-service"..." type=io.containerd.service.v1 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." type=io.containerd.monitor.v1 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." type=io.containerd.runtime.v1 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.service.v1.tasks-service"..." type=io.containerd.service.v1 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." type=io.containerd.grpc.v1 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." type=io.containerd.grpc.v1 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." type=io.containerd.grpc.v1 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." type=io.containerd.grpc.v1 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." type=io.containerd.grpc.v1 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." type=io.containerd.grpc.v1 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." type=io.containerd.grpc.v1 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." type=io.containerd.grpc.v1 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." type=io.containerd.grpc.v1 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." type=io.containerd.grpc.v1 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." type=io.containerd.grpc.v1 
time="2019-02-04T16:00:25+01:00" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." type=io.containerd.grpc.v1 
time="2019-02-04T16:00:25+01:00" level=info msg=serving... address="/var/run/docker/containerd/docker-containerd-debug.sock" 
time="2019-02-04T16:00:25+01:00" level=info msg=serving... address="/var/run/docker/containerd/docker-containerd.sock" 
time="2019-02-04T16:00:25+01:00" level=info msg="containerd successfully booted in 0.008216s" 
time="2019-02-04T16:00:25.884102534+01:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4203c57b0, READY" module=grpc
time="2019-02-04T16:00:25.887492664+01:00" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2019-02-04T16:00:25.887505949+01:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2019-02-04T16:00:25.887530644+01:00" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///var/run/docker/containerd/docker-containerd.sock 0  <nil>}]" module=grpc
time="2019-02-04T16:00:25.887540195+01:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2019-02-04T16:00:25.887563400+01:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4201821f0, CONNECTING" module=grpc
time="2019-02-04T16:00:25.887660990+01:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4201821f0, READY" module=grpc
time="2019-02-04T16:00:25.909326944+01:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
time="2019-02-04T16:00:25.909459182+01:00" level=warning msg="Your kernel does not support cgroup memory limit"
time="2019-02-04T16:00:25.909468854+01:00" level=warning msg="Unable to find cpu cgroup in mounts"
time="2019-02-04T16:00:25.909473921+01:00" level=warning msg="Unable to find blkio cgroup in mounts"
time="2019-02-04T16:00:25.909478263+01:00" level=warning msg="Unable to find cpuset cgroup in mounts"
time="2019-02-04T16:00:25.909552255+01:00" level=warning msg="mountpoint for pids not found"
Error starting daemon: Devices cgroup isn't mounted

Is there any technique to start docker daemon in a chroot Jail ?

Answer 1

the chroot needs cgroups - you have to ensure the chroot has the correct file systems mounted /proc /dev /sys, which you can either mount again or rbind from host - then you need to mount the cgroups inside the jail (if you have rbinded /sys you might already have them in /sys/fs/cgroup). You can use these scripts to mount cgroups. - then you start docker

Since you can't load kernel modules from a chroot you have to ensure those are mounted beforehand, like overlay for the storage driver and nf_conntrack for networking. Alternatively you can always fallback to the vfs storage driver and disabling iptables...