systemd-nspawn
systemd-nspawn is like the chroot command, but it is a chroot on steroids.
systemd-nspawn may be used to run a command or OS in a light-weight namespace container. It is more powerful than chroot since it fully virtualizes the file system hierarchy, as well as the process tree, the various IPC subsystems and the host and domain name.
systemd-nspawn limits access to various kernel interfaces in the container to read-only, such as /sys, /proc/sys or /sys/fs/selinux. Network interfaces and the system clock may not be changed from within the container. Device nodes may not be created. The host system cannot be rebooted and kernel modules may not be loaded from within the container.
systemd-nspawn is a simpler tool to configure than LXC or Libvirt.
Installation
systemd-nspawn is part of and packaged with systemd.
Examples
Create and boot a minimal Arch Linux container
First install .
Next, create a directory to hold the container. In this example we will use .
Next, we use pacstrap to install a basic Arch system into the container. At minimum we need to install the package.
# pacstrap -K -c ~/MyContainer base [additional packages/groups]
Once your installation is finished, chroot into the container, and set a root password:
# systemd-nspawn -D ~/MyContainer # passwd # logout
Finally, boot into the container:
# systemd-nspawn -b -D ~/MyContainer
The -b option will boot the container (i.e. run systemd as PID=1), instead of just running a shell, and specifies the directory that becomes the container's root directory.
After the container starts, log in as "root" with your password.
The container can be powered off by running from within the container. From the host, containers can be controlled by the machinectl tool.
Create a Debian or Ubuntu environment
Install , and one or both of or depending on which distribution you want.
From there it is rather easy to set up Debian or Ubuntu environments:
# cd /var/lib/machines # debootstrap --include=systemd-container --components=main,universe codename container-name repository-url
For Debian, valid code names are either the rolling names like "stable" and "testing" or release names like "stretch" and "sid". For Ubuntu, the code name like "xenial" or "zesty" should be used. A complete list of code names is in and the official table of code names to version numbers can be found in . In case of a Debian image the "repository-url" can be https://deb.debian.org/debian/. For an Ubuntu image, the "repository-url" can be http://archive.ubuntu.com/ubuntu/. "repository-url" should not contain a trailing slash.
Just like Arch, Debian and Ubuntu will not let you log in without a password. To set the root password, run systemd-nspawn without the -b option:
# cd /var/lib/machines # systemd-nspawn -D ./container-name # passwd # logout
Build and test packages
See Creating packages for other distributions for example uses.
Management
Containers located in /var/lib/machines/ can be controlled by the machinectl command, which internally controls instances of the unit. The subdirectories in /var/lib/machines/ correspond to the container names, i.e. .
/var/lib/machines/ for some reason, it can be symlinked. See machinectl(1) § FILES AND DIRECTORIES for details.Default systemd-nspawn options
It is important to realize that containers started via machinectl or use different default options than containers started manually by the systemd-nspawn command. The extra options used by the service are:
-b/ – Managed containers automatically search for an init program and invoke it as PID 1.- which implies – Managed containers get a virtual network interface and are disconnected from the host network. See #Networking for details.
- – Managed containers use the feature by default if supported by the kernel. See #Unprivileged containers for implications.
The behaviour can be overridden in per-container configuration files, see #Configuration for details.
machinectl
Containers can be managed by the machinectl subcommand container-name command. For example, to start a container:
$ machinectl start container-name
Similarly, there are subcommands such as , , and . See for detailed explanations.
Other common commands are:
machinectl list– show a list of currently running containers- – open an interactive login session in a container
- – open an interactive shell session in a container (this immediately invokes a user process without going through the login process in the container)
machinectl enable container-nameand – enable or disable a container to start at boot, see #Enable container to start at boot for details
machinectl also has subcommands for managing container (or virtual machine) images and image transfers. See and for details.
systemd toolchain
Much of the core systemd toolchain has been updated to work with containers. Tools that do usually provide a option which will take a container name as argument.
Examples:
See journal logs for a particular machine:
# journalctl -M container-name
Show control group contents:
$ systemd-cgls -M container-name
See startup time of container:
$ systemd-analyze -M container-name
For an overview of resource usage:
$ systemd-cgtop
Configuration
Per-container settings
To specify per-container settings and not global overrides, the .nspawn files can be used. See for details.
Enable container to start at boot
When using a container frequently, you may want to start it at boot.
First make sure that the machines.target is enabled.
Containers discoverable by machinectl can be enabled or disabled:
$ machinectl enable container-name
Resource control
You can take advantage of control groups to implement limits and resource management of your containers with , see systemd.resource-control(5). For example, you may want to limit the memory amount or CPU usage. To limit the memory consumption of your container to 2 GiB:
# systemctl set-property systemd-nspawn@container-name.service MemoryMax=2G
Or to limit the CPU time usage to roughly the equivalent of 2 cores:
# systemctl set-property systemd-nspawn@container-name.service CPUQuota=200%
This will create permanent files in .
According to the documentation, is the preferred method to keep in check memory consumption, but it will not be hard-limited as is the case with . You can use both options leaving as the last line of defense. Also take in consideration that you will not limit the number of CPUs the container can see, but you will achieve similar results by limiting how much time the container will get at maximum, relative to the total CPU time.
Networking
systemd-nspawn containers can use either host networking or private networking:
- In the host networking mode, the container has full access to the host network. This means that the container will be able to access all network services on the host and packets coming from the container will appear to the outside network as coming from the host (i.e. sharing the same IP address).
- In the private networking mode, the container is disconnected from the host's network. This makes all network interfaces unavailable to the container, with the exception of the loopback device and those explicitly assigned to the container. There is a number of different ways to set up network interfaces for the container:
- an existing interface can be assigned to the container (e.g. if you have multiple Ethernet devices),
- a virtual network interface associated with an existing interface (i.e. VLAN interface) can be created and assigned to the container,
- a virtual Ethernet link between the host and the container can be created.
- In the latter case the container's network is fully isolated (from the outside network as well as other containers) and it is up to the administrator to configure networking between the host and the containers. This typically involves creating a network bridge to connect multiple (physical or virtual) interfaces or setting up a Network Address Translation between multiple interfaces.
The host networking mode is suitable for application containers which do not run any networking software that would configure the interface assigned to the container. Host networking is the default mode when you run systemd-nspawn from the shell.
On the other hand, the private networking mode is suitable for system containers that should be isolated from the host system. The creation of virtual Ethernet links is a very flexible tool allowing to create complex virtual networks. This is the default mode for containers started by machinectl or .
The following subsections describe common scenarios. See for details about the available systemd-nspawn options.
Use host networking
To disable private networking and the creation of a virtual Ethernet link used by containers started with machinectl, add a .nspawn file with the following option:
This will override the -n/ option used in and the newly started containers will use the host networking mode.
Use a virtual Ethernet link
If a container is started with the -n/ option, systemd-nspawn will create a virtual Ethernet link between the host and the container. The host side of the link will be available as a network interface named . The container side of the link will be named . Note that this option implies .
- If the container name is too long, the interface name will be shortened (e.g.
ve-long-conKQGhinstead ofve-long-container-name) to fit into the 15-characters limit. The full name will be set as thealtnameproperty of the interface (see ip-link(8)) and can be still used to reference the interface. - When examining the interfaces with
ip link, interface names will be shown with a suffix, such asve-container-name@if2andhost0@if9. The@ifNis not actually part of the interface name; instead,ip linkappends this information to indicate which "slot" the virtual Ethernet cable connects to on the other end.
- For example, a host virtual Ethernet interface shown as
ve-foo@if2is connected to the containerfoo, and inside the container to the second network interface – the one shown with index 2 when runningip linkinside the container. Similarly, the interface namedhost0@if9in the container is connected to the 9th network interface on the host.
When you start the container, an IP address has to be assigned to both interfaces (on the host and in the container). If you use systemd-networkd on the host as well as in the container, this is done out-of-the-box:
- the file on the host matches the interface and starts a DHCP server, which assigns IP addresses to the host interface as well as the container,
- the file in the container matches the interface and starts a DHCP client, which receives an IP address from the host.
If you do not use systemd-networkd, you can configure static IP addresses or start a DHCP server on the host interface and a DHCP client in the container. See Network configuration for details.
To give the container access to the outside network, you can configure NAT as described in Internet sharing#Enable NAT. If you use systemd-networkd, this is done (partially) automatically via the option in . However, this issues just one iptables (or nftables) rule such as
-t nat -A POSTROUTING -s 192.168.163.192/28 -j MASQUERADE
The table has to be configured manually as shown in Internet sharing#Enable NAT. You can use a wildcard to match all interfaces starting with :
# iptables -A FORWARD -i ve-+ -o internet0 -j ACCEPT
Additionally, you need to open the UDP port 67 on the interfaces for incoming connections to the DHCP server (operated by systemd-networkd):
# iptables -A INPUT -i ve-+ -p udp -m udp --dport 67 -j ACCEPT
Use a network bridge
If you have configured a network bridge on the host system, you can create a virtual Ethernet link for the container and add its host side to the network bridge. This is done with the --network-bridge=bridge-name option. Note that implies , i.e. the virtual Ethernet link is created automatically. However, the host side of the link will use the prefix instead of , so the systemd-networkd options for starting the DHCP server and IP masquerading will not be applied.
The bridge management is left to the administrator. For example, the bridge can connect virtual interfaces with a physical interface, or it can connect only virtual interfaces of several containers. See systemd-networkd#Network bridge with DHCP and systemd-networkd#Network bridge with static IP addresses for example configurations using systemd-networkd.
There is also a --network-zone=zone-name option which is similar to but the network bridge is managed automatically by systemd-nspawn and systemd-networkd. The bridge interface named is automatically created when the first container configured with --network-zone=zone-name is started, and is automatically removed when the last container configured with --network-zone=zone-name exits. Hence, this option makes it easy to place multiple related containers on a common virtual network. Note that interfaces are managed by systemd-networkd same way as interfaces using the options from the file.
Use a "macvlan" or "ipvlan" interface
Instead of creating a virtual Ethernet link (whose host side may or may not be added to a bridge), you can create a virtual interface on an existing physical interface (i.e. VLAN interface) and add it to the container. The virtual interface will be bridged with the underlying host interface and thus the container will be exposed to the outside network, which allows it to obtain a distinct IP address via DHCP from the same LAN as the host is connected to.
systemd-nspawn offers 2 options:
- – the virtual interface will have a different MAC address than the underlying physical and will be named
mv-interface. - – the virtual interface will have the same MAC address as the underlying physical and will be named .
Both options imply .
Use an existing interface
If the host system has multiple physical network interfaces, you can use the to assign to the container (and make it unavailable to the host while the container is started). Note that implies .
Port mapping
When private networking is enabled, individual ports on the host can be mapped to ports on the container using the / option or by using the setting in an .nspawn file. This is done by issuing iptables rules to the table, but the FORWARD chain in the table needs to be configured manually as shown in #Use a virtual Ethernet link.
For example, to map a TCP port 8000 on the host to the TCP port 80 in the container:
Domain name resolution
Domain name resolution in the container can be configured the same way as on the host system. Additionally, systemd-nspawn provides options to manage the file inside the container:
- can be used on command-line
- can be used in .nspawn files
These corresponding options have many possible values which are described in . The default value is , which means that:
- If is enabled, the is left as it is in the container.
- Otherwise, if systemd-resolved is running on the host, its stub file is copied or bind-mounted into the container.
- Otherwise, the file is copied or bind-mounted from the host to the container.
In the last two cases, the file is copied, if the container root is writeable, and bind-mounted if it is read-only.
For the second case where systemd-resolved runs on the host, systemd-nspawn expects it to also run in the container, so that the container can use the stub symlink file from the host. If not, the default value no longer works, and you should replace the symlink by using one of the replace-* options.
Tips and tricks
Running non-shell/init commands
From :
- "[The option]
--as-pid2[invokes] the shell or specified program as process ID (PID) 2 instead of PID 1 (init). [...] It is recommended to use this mode to invoke arbitrary commands in containers, unless they have been modified to run correctly as PID 1. Or in other words: this switch should be used for pretty much all commands, except when the command refers to an init or shell implementation."
Unprivileged containers
systemd-nspawn supports unprivileged containers, though the containers need to be booted as root.
The easiest way to do this is to let systemd-nspawn automatically choose an unused range of UIDs/GIDs by using the option:
# systemd-nspawn -bUD ~/MyContainer
If kernel supports user namespaces, the option is equivalent to . See for details.
If a container has been started with a private UID/GID range using the option (or on a filesystem where requires ), you need to keep using it that way to avoid permission errors. Alternatively, it is possible to undo the effect of on the container's file system by specifying a range of IDs starting at 0:
# systemd-nspawn -D ~/MyContainer --private-users=0 --private-users-ownership=chown
Use an X environment
See Xhost and Change root#Run graphical applications from chroot.
You will need to set the DISPLAY environment variable inside your container session to connect to the external X server.
X stores some required files in the directory. In order for your container to display anything, it needs access to those files. To do so, append the option when starting the container.
Avoiding xhost
xhost only provides rather coarse access rights to the X server. More fine-grained access control is possible via the file. Unfortunately, just making the file accessible in the container will not do the job:
your file is specific to your host, but the container is a different host.
The following trick adapted from stackoverflow can be used to make your X server accept the file from an X application run inside the container:
$ XAUTH=/tmp/container_xauth $ xauth nextract - "$DISPLAY" | sed -e 's/^..../ffff/' | xauth -f "$XAUTH" nmerge - # systemd-nspawn -D myContainer --bind=/tmp/.X11-unix --bind="$XAUTH" -E DISPLAY="$DISPLAY" -E XAUTHORITY="$XAUTH" --as-pid2 /usr/bin/xeyes
The second line above sets the connection family to "FamilyWild", value , which causes the entry to match every display. See for more information.
Using X nesting/Xephyr
Another simple way to run X applications and avoid the risks of a shared X desktop is using X nesting. The advantages here are avoiding interaction between in-container applications and non-container applications entirely and being able to run a different desktop environment or window manager, the downsides are less performance and the lack of hardware acceleration when using Xephyr.
Start Xephyr outside of the container using:
# Xephyr :1 -resizeable
Then start the container with the following options:
--setenv=DISPLAY=:1 --bind-ro=/tmp/.X11-unix/X1
No other binds are necessary.
You might still need to manually set in the container under some circumstances (mostly if used with -b).
Run Firefox
# systemd-nspawn --setenv=DISPLAY=:0 \
--setenv=XAUTHORITY=~/.Xauthority \
--bind-ro=$HOME/.Xauthority:/root/.Xauthority \
--bind=/tmp/.X11-unix \
-D ~/containers/firefox \
--as-pid2 \
firefox
Alternatively you can boot the container and let e.g. systemd-networkd set up the virtual network interface:
# systemd-nspawn --bind-ro=$HOME/.Xauthority:/root/.Xauthority \
--bind=/tmp/.X11-unix \
-D ~/containers/firefox \
--network-veth -b
Once your container is booted, run the Xorg binary like so:
# systemd-run -M firefox --setenv=DISPLAY=:0 firefox
3D graphics acceleration
To enable accelerated 3D graphics, it may be necessary to bind mount to the container by adding the following line to the .nspawn file:
Bind=/dev/dri
The above trick was adopted from patrickskiba.com. This notably solves the problem of
libGL error: MESA-LOADER: failed to retrieve device information libGL error: Version 4 or later of flush extension not found libGL error: failed to load driver: i915
You can confirm that the it has been enabled by running glxinfo or .
Nvidia GPUs
If you cannot install the same nvidia driver version on the container as on the host, you may need to also bind the driver library files. You can run on the host to see all the files it contains. You do not need to copy everything over. The following systemd override file will bind all the necessary files over when the container is run via .
Access host filesystem
See and in .
If both the host and the container are Arch Linux, then one could, for example, share the pacman cache:
# systemd-nspawn --bind=/var/cache/pacman/pkg
Or you can specify per-container bind using the file:
/etc/systemd/nspawn/''my-container''.nspawn
[Files] Bind=/var/cache/pacman/pkg
To bind the directory to a different path within the container, add the path be separated by a colon. For example:
# systemd-nspawn --bind=/path/to/host_dir:/path/to/container_dir
In case of #Unprivileged containers, the resulting mount points will be owned by the nobody user. This can be modified with the mount option:
# systemd-nspawn --bind=/path/to/host_dir:/path/to/container_dir:idmap
Run on a non-systemd system
See Init#systemd-nspawn.
Use Btrfs subvolume as container root
To use a Btrfs subvolume as a template for the container's root, use the flag. This takes a snapshot of the subvolume and populates the root directory for the container with it.
For example, to use a snapshot located at :
# systemd-nspawn --template=/.snapshots/403/snapshots -b -D my-container
where is the name of the directory that will be created for the container. After powering off, the newly created subvolume is retained.
Use temporary Btrfs snapshot of container
One can use the --ephemeral or flag to create a temporary btrfs snapshot of the container and use it as the container root. Any changes made while booted in the container will be lost. For example:
# systemd-nspawn -D my-container -xb
where my-container is the directory of an existing container or system. For example, if is a btrfs subvolume one could create an ephemeral container of the currently running host system by doing:
# systemd-nspawn -D / -xb
After powering off the container, the btrfs subvolume that was created is immediately removed.
Run docker in systemd-nspawn
Since Docker 20.10, it is possible to run Docker containers inside an unprivileged systemd-nspawn container with cgroups v2 enabled (default in Arch Linux) without undermining security measures by disabling cgroups and user namespaces. To do so, edit (create if absent) and add the following configurations.
/etc/systemd/nspawn/myContainer.nspawn
[Exec] SystemCallFilter=add_key keyctl bpf
Then, Docker should work as-is inside the container.
Since overlayfs does not work with user namespaces and is unavailable inside systemd-nspawn, by default, Docker falls back to using the inefficient vfs as its storage driver, which creates a copy of the image each time a container is started. This can be worked around by using fuse-overlayfs as its storage driver. To do so, we need to first expose fuse to the container:
and then allow the container to read and write the device node:
# systemctl set-property systemd-nspawn@myContainer DeviceAllow='/dev/fuse rwm'
Finally, install the package inside the container. You need to restart the container for all the configuration to take effect.
Troubleshooting
Root login fails
If you get the following error when you try to login (i.e. using ):
arch-nspawn login: root Login incorrect
And the journal shows:
pam_securetty(login:auth): access denied: tty 'pts/0' is not secure !
It is possible to either delete and on the container file system, or simply add the desired pty terminal devices (like ), as necessary, to on the container file system. Any changes will be overridden on the next boot, therefore it is necessary to also remove the entry from on the container file system, see . If you opt for deletion, you might also optionally blacklist the files (NoExtract) in /etc/pacman.conf to prevent them from getting reinstalled. See for details.
execv(...) failed: Permission denied
When trying to boot the container via (or executing something in the container), and the following error comes up:
execv(/usr/lib/systemd/systemd, /lib/systemd/systemd, /sbin/init) failed: Permission denied
even though the permissions of the files in question (i.e. ) are correct, this can be the result of having mounted the file system on which the container is stored as non-root user. For example, if you mount your disk manually with an entry in fstab that has the options , systemd-nspawn will not allow executing the files even if they are owned by root.
Terminal type in TERM is incorrect (broken colors)
When logging into the container via machinectl login, the colors and keystrokes in the terminal within the container might be broken. This may be due to an incorrect terminal type in environment variable. The environment variable is not inherited from the shell on the host, but falls back to a default fixed in systemd (), unless explicitly configured. To configure, within the container create a configuration overlay for the systemd service that launches the login getty for machinectl login, and set to the value that matches the host terminal you are logging in from:
Alternatively use . It properly inherits the environment variable from the terminal.
Mounting a NFS share inside the container
Not possible at this time (June 2019).