In discussions around the ongoing transition from X.org to Wayland, I regularly come across comments along the lines of "Linux security is not there yet". This refers to Wayland's promise of better (but not perfect) desktop isolation (e.g. preventing keylogging between two GUI applications).
This implies to me that display managers on other operating systems (Windows, macOS, Android, iOS) have better isolation than both X.org and Wayland? Is this true?
Microsoft Windows, since Vista, has at least attempted to isolate windows, for example using UIPI and session 0 isolation. This is not perfect, and there are some limitations. Windows exposes many large APIs to client windows, making it possible to bypass isolation. This is a particular problem for UAC consent prompts, where the assumption is that only a genuine user can grant consent. Additionally, AppContainer is able to isolate individual windows. Unfortunately, I know little about how Windows manages GUI isolation, and surely the state of things has changed since Vista. What is true though is that Windows does attempt to isolate individual windows to a certain extent, but there are bypasses.
Xorg is one of the original display managers for Linux, having superseded Xfree86. It uses the X11 protocol. When a graphical application wishes to draw to the screen, it connects to the X server through a UNIX domain socket or through the local network, using the X11 protocol. To authenticate to the X server, it must provide the X11 cookie, which is a random value in a file readable only by a single user, or exported as an environmental variable for that user. Any program that has access to the X11 cookie has full access to the X server. It is able to read keystrokes, inject keystrokes, take screenshots, move windows, etc. As such, graphical isolation under Xorg is limited to DAC (UNIX permissions) against the file containing the X11 cookie. Given that we often (foolishly) enter our root passwords in an X11 session at the same time our possibly vulnerable browser has a hundred untrusted tabs open, this security design is not the best idea. Attempts to improve X11 security, such as the XSECURITY extension, have failed miserably. It is too complex, and too easy to bypass.
There is a way to get graphical isolation on systems with X11, though. Creating a nested X server, for example with Xephyr, allows starting individual applications in their own X server. Combining this with mandatory access controls to force each application to only be able to read its own nested X server's X11 cookie completes this. This provides isolation, but does nothing for the excessive complexity of the X11 protocol and its extensions. Vulnerabilities in the decades-old application continue to pop up.
Wayland is supposed to be the answer to Linux desktop security. It integrates itself with the window manager in the form of a shared library, allowing the window manager to be able to manage permissions for individual windows. Each window can request permissions from the window manager, and the window manager can grant or refuse them. The overall architecture for Wayland is vastly different from Xorg, and there are many internal differences not directly related to security (for example, Wayland does not do rendering on its own, rather it requires the client window do the rendering). The differences alone would take up an entire answer. Security-wise, Wayland is superior in that it provides confidentiality, integrity, and availability to a desktop full of untrusted graphical applications, and it runs using a simpler protocol, requiring fewer privileges to function properly.
I know little about mobile computing. One big difference between mobile operating systems of all types and desktop operating systems is that mobile devices tend to be more monolithic and limited in their configurability. This makes security policies easier to design. As a result, inter-window communication becomes less important, to the point that Android can even run each individual application under its own UID and GID, completely sandboxed away from the rest of the system. iOS likely does something similar. If I discover more about Mac OSX (which is certainly not fully isolated from malicious applications) and Android/iOS graphical isolation, I will edit this answer.
To answer your question, many operating systems do have better isolation than Xorg (which has none), but not better than Wayland which, currently, is not known to be possible to bypass short of using novel exploits or hijacking the entire user. Wayland is a major improvement over the X11 model.
