If a malicious application has found its way onto a computer running a localhost only WebSocket server, would it help if the communications used SSL?
If so, how could SSL fail if a malicious app
SSL will usually not help.
If the attacker has complete control over the computer, then he wins. He can inspect and modify the memory of all process in the computer.
If the attacker is only a non-administrative user, then he won't be able to spy on process from other users. However, if the attacker can run code as the same user as either the server or the client of your envisioned localhost-only connection (or both), then he can plunder the RAM of the involved process, and he wins. Also, assuming a non-administrative attacker means that you trust that the said attacker could enter the machine but did not find any privilege-escalation exploitable hole: that's a rather optimistic assumption.
If the attacker can only listen to communications, but not have a look at the memory of the process, then SSL may help, since it will prevent spying. However, this is not a very realistic scenario: spying on localhost connections requires local admin privileges.
Where SSL may help is about authentication: a local attacker, with non-admin rights, may try to connect to the server and impersonate the client, or vice-versa. SSL provides authentication: the client validates the server's certificate, and, if the server asked for a client certificate, the server validates the client's certificate. Note, though, that on localhost, there are more efficient ways for a server to know who is at the other end of the socket (e.g. getpeereid()).
