Is systemd's hand-rolled Desktop-Bus-over-SSH tunnel a security worry?

Question

On 2017-09-02 Peter Wullinger wrote:

☑ tunneled connections to remote dbus
☑ hardcode ssh (without path)
☑ have escape, parse, unescape bug
☑ don't document any of this

When one uses the -H or --host option to the systemctl program, systemd tells its D-Bus library to execute ssh under the covers running a program named systemd-stdio-bridge on the remote machine. That program acts as a simple message pump between the D-Bus broker on the remote machine and systemctl, the D-Bus client, on the local machine. systemctl just talks to the Desktop Bus that is being proxied over SSH and systemd-stdio-bridge, as if it were talking to the local system one.

In response, other people wondered that this might be another systemd security problem. Is it?

(This is not a question about SSH security in general, but rather with respect to this specific systemd mechanism. There are other questions to look at for general SSH security, such as "What are the security implications of SSH tunneling?". Nor is this a question like "What are the security implications of systemd compared to systemv init?". For starters, this is not in any way about van Smoorenburg rc files.)

Answer 1

Discussing this on Twitter is the modern day equivalent of Fermat discussing mathematical proofs via marginalia. So let us expand upon what Peter Wullinger did not have the space to fit into a margin a Twitter post.

There were, of course, other ways that the systemd people could have done this:

• Advise system administrators to use

  ssh -x -- account@host systemctl stuff…

  instead of

  systemctl -H account@host stuff…

• Use OpenSSH's built in forwarding mechanism for local-domain sockets so that internally systemctl spawns something like

  ssh -xT -L /run/user/user/account@host/bus:/run/dbus/system_bus_socket -- account@host /bin/true

  instead of (as it does)

  ssh -xT -- account@host systemd-stdio-bridge

There are a number of problems with the approach actually taken, several of which are security-related. None appear, so far, to rise to the level of outright vulnerabilities, however.

"What's this undocumented systemd-stdio-bridge program?"

Consider, first off, the system administrator looking at logs and the output of ps, and wondering about these people all running systemd-stdio-bridge via SSH. We have, as can be seen from "Strange 'agetty' process running on my VPS.", people who wonder about Weitse Venema's agetty program and we know that MacOS already has malware named systemd in an attempt to fool people, so it is not a stretch nor unreasonable to imagine that system administrators would wonder about systemd-stdio-bridge too.

The problem is that this is another occasion where systemd's doco is worse than poor: it is outright absent. There is no manual page for systemd-stdio-bridge written by the systemd people, at all. So the system administrator's first recourse, reading the manual, is no help. (This is intentional on the parts of the systemd people, who have not included system administrators wanting to identify running programs in their use case.)

It might help the system administrator to learn that the program went through a phase of being called systemd-bus-proxyd, which had a manual page. Both it and the manual page were removed from systemd, but when the older program was later brought back by systemd developer David Herrmann, a manual page was not restored along with it. (Note, if you do go and dig it up, that the old manual page for systemd-bus-proxyd does not correctly describe the current command-line usage of systemd-stdio-bridge.)

Had the systemd developers considered the doco alongside the program, they might have spotted one current bug: systemctl in certain circumstances passes options that systemd-stdio-bridge does not support, and systemd-stdio-bridge supports different options that systemctl makes no use of.

"What's this undocumented sd_bus_set_address stuff?"

One of Peter Wullinger's notes is that none of this is documented, and this is at least alas true in the case of the system administrator who tries to resort to reading the code for the systemd-stdio-bridge program. This is a compiled C program, not one written in an interpreted scripting language. As Peter Wullinger points out, it relies upon calling a sd_bus_set_address() function. The systemd people provide no manual page for sd_bus_set_address(), leaving dangling hyperlinks on other systemd manual pages (such as the hyperlink from the sd_bus_new() manual page for example).

Parsing bugs

It was not initially clear what Peter Wullinger thinks to be an "escape, parse, unescape bug". But there is a lot of parsing going on under the covers; not the least of which is systemd's Desktop Bus library constructing strings such as "unixexec:path=ssh,argv1=-xT,argv2=--,argv3=account%40host,argv4=systemd-stdio-bridge,arg5=--machine=fred" from their constituent parts in one place only to go and parse them back into their constituent parts in another. And we know from Daniel J. Bernstein (amongst many others) that designing to avoid parsing, including this sort of marshalling into pseudo-human-readable form and then unmarshalling back into machine-readable form, is a good idea.

It turns out that Peter Wullinger was referring to the reason that argv2=-- is now in there. That is of course a well known problem, that is sad to say exemplified far too widely. However, an additional and more subtle parsing-related problem in the aforementioned superfluous serialization and deserialization is the very ad hoc nature of the system. Notice that an equals sign character to be passed in argv3 is escaped as %3d. The equals sign in arg5=--machine= is however not escaped. The serialization does not obey the same rules as the deserialization. This way, bugs and security problems lie; and this is one of the very things that Daniel J. Bernstein recommended to avoid by not parsing in the first place.

Reinventing the wheel

There is, of course the notion that systemd-stdio-bridge is a reinvention of the wheel that has not knocked all of the corners off to make it round enough, yet. After all, as aforementioned, OpenSSH has a built-in mechanism, far more widely used and tested, for forwarding access to local-domain sockets over the SSH connection from remote host to local machine. (It has had it since OpenSSH 6.7, released the year before systemd developer David Herrmann reintroduced the systemd-stdio-bridge program. Patches for this have been around since 2006, years before systemd existed.)

Militating against this are two things.

First: Spawning a program remotely and then talking to it via its standard input and output, transmitted over the SSH connection, is not exactly an unusual use of SSH. It's the rexec model of doing things, after all.

Second: Local-domain socket forwarding has to be done carefully. With the standard I/O mechanism, no other D-Bus client program can (absent tricks like debug access) use the forwarding mechanism to itself connect to the remote D-Bus broker. The aforementioned mechanism however has to be careful to:

• not allow anyone other than the user any access at all to /run/user/user/account@host/ so that only the user's (and the superuser's) processes can access the local proxy socket
• guard against /run/user/user/ being writable by someone other than the user, so that no-one else can come along and sneakily replace /run/user/user/account@host with a symbolic link or perform other such tricks
• take care that /run/user/user/account@host/bus does not collide amongst multiple invocations of systemctl (which can be done by employing a locally unique name instead of just bus)

Further reading

• V.R. (2015-10-11). "Parsing in critical paths". Structural and semantic deficiencies in the systemd architecture for real-world service management, a technical treatise. darknedgy.
• chromatic (2010-07-10). "Don't Parse That String!". Modern Perl Programming.
• William Ahern. OpenSSH Unix Domain Socket Forwarding. 25thandclement.

Answer 2

To elaborate on a short tweet made after digging through a pile of systemd documentation and source code (none of them pleasant, which might explain the tone).

The whole thing was triggered by this tweet, which indicates that systemd does not (as noted by the main developer multiple times) validate all input.

systemctl --host <host> is documented to execute the specified operation remotely via ssh.

How this works is

1. systemctl passes the host parameter down to sd_bus_open_system_remote(), which --as an undocumented feature (bug report)-- also supports <hostspec>:<machine> notation to tell it to connect to a bus on a remote host in a given container.

2. sd_bus_open_system_remote() constructs a special bus address unixexec:path=ssh,argv1=-xT,argv2=--,argv3=<hostname>,argv4=systemd-stdio-bridge <container>. Note that this picks ssh from $PATH and systemd-stdio-bridge from the remote $PATH of the target user on the target machine. This special bus address seems to have been added to the dbus specification and reference-implementation by one of the systemd authors himself.

3. This string gets parsed (still locally) by bus_start_address() which has support for arbitrary dbus proxy commands via unixexec.

4. And finally, this triggers bus_start_address() to execute the provided command line and do dbus via a pipe to stdin/stdout of that program.

On my count, that's three layers of undocumented indirection to get a simple convenience feature that you'll also get with ssh <host> systemctl --machine <machine>.

Such implicit magic usually gets my internal bug searcher on autopilot.

Well, there was one, but it's not critical: Before a recent commit, systemctl would strip everything past the first : from the hostname, but pass everything else to ssh as the first command line argument. This is still executed locally and pretty obvious from the command line, so it doesn't seem that much of a problem.

But modulo the convoluted nature of it all, having a remote proxy is probably one of the more secure methods of handling this. -L is problematic, since it requires you to enable SSH port forwarding and proper cleanup and proper security for the local sockets. -W does not support unix domain sockets (as of now). At the very least the current method allows you to use ForceCommand.

The real security threat probably lies in the nature of the whole process replacing something that can be done explicitly with a largely undocumented, automated process that also hardcodes quite a few assumptions on the way.