0

Some weeks ago ssh login was no longer possible to a remote Mac. Problem started occuring around upgrading Webmin to version 1.550 and/or changing settings using webmin.

Using: * Mac OS X 10.6.5 + Webmin 1.550 + Virtualmin 3.66GPL

Expected Results: No problems logging in using ssh.

Actual Results: ssh: connect to host host.domain.tld port 22: Operation timed out

Regression: After more investigation it appears that:

  1. sshd is correctly started when it is configured to not detach (-D), using $ sudo /usr/sbin/sshd -D -d -d -d -e
debug2: load_server_config: filename /etc/sshd_config
debug2: load_server_config: done config len = 493
debug2: parse_server_config: config /etc/sshd_config len 493
debug3: /etc/sshd_config:14 setting Protocol 2
debug3: /etc/sshd_config:30 setting SyslogFacility AUTHPRIV
debug3: /etc/sshd_config:32 setting LogLevel DEBUG3
debug3: /etc/sshd_config:39 setting MaxAuthTries 3
debug3: /etc/sshd_config:108 setting UseDNS no
debug3: /etc/sshd_config:111 setting MaxStartups 5
debug3: /etc/sshd_config:119 setting Subsystem sftp /usr/libexec/sftp-server
debug3: /etc/sshd_config:121 setting IgnoreRhosts yes
debug3: /etc/sshd_config:122 setting IgnoreUserKnownHosts no
debug3: /etc/sshd_config:123 setting PrintMotd yes
debug3: /etc/sshd_config:124 setting StrictModes yes
debug3: /etc/sshd_config:125 setting RSAAuthentication yes
debug3: /etc/sshd_config:126 setting PermitEmptyPasswords no
debug3: /etc/sshd_config:127 setting PasswordAuthentication yes
debug3: /etc/sshd_config:128 setting DenyGroups deniedssh
debug3: /etc/sshd_config:129 setting PubkeyAuthentication yes
debug3: /etc/sshd_config:130 setting GatewayPorts no
debug3: /etc/sshd_config:131 setting AllowTcpForwarding yes
debug3: /etc/sshd_config:132 setting KeepAlive yes
debug1: sshd version OpenSSH_5.2p1
debug3: Not a RSA1 key file /etc/ssh_host_rsa_key.
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug3: Not a RSA1 key file /etc/ssh_host_dsa_key.
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-D'
debug1: rexec_argv[2]='-d'
debug1: rexec_argv[3]='-d'
debug1: rexec_argv[4]='-d'
debug1: rexec_argv[5]='-e'
debug2: fd 3 setting O_NONBLOCK
debug1: Bind to port 22 on ::.
Server listening on :: port 22.
debug2: fd 4 setting O_NONBLOCK
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug1: fd 5 clearing O_NONBLOCK
debug1: Server will not fork when running in debugging mode.
debug3: send_rexec_state: entering fd = 8 config len 493
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug3: recv_rexec_state: entering fd = 5
debug3: ssh_msg_recv entering
debug3: recv_rexec_state: done
debug2: parse_server_config: config rexec len 493
debug3: rexec:14 setting Protocol 2
debug3: rexec:30 setting SyslogFacility AUTHPRIV
debug3: rexec:32 setting LogLevel DEBUG3
debug3: rexec:39 setting MaxAuthTries 3
debug3: rexec:108 setting UseDNS no
debug3: rexec:111 setting MaxStartups 5
debug3: rexec:119 setting Subsystem sftp    /usr/libexec/sftp-server
debug3: rexec:121 setting IgnoreRhosts yes
debug3: rexec:122 setting IgnoreUserKnownHosts no
debug3: rexec:123 setting PrintMotd yes
debug3: rexec:124 setting StrictModes yes
debug3: rexec:125 setting RSAAuthentication yes
debug3: rexec:126 setting PermitEmptyPasswords no
debug3: rexec:127 setting PasswordAuthentication yes
debug3: rexec:128 setting DenyGroups deniedssh
debug3: rexec:129 setting PubkeyAuthentication yes
debug3: rexec:130 setting GatewayPorts no
debug3: rexec:131 setting AllowTcpForwarding yes
debug3: rexec:132 setting KeepAlive yes
debug1: sshd version OpenSSH_5.2p1
debug3: Not a RSA1 key file /etc/ssh_host_rsa_key.
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug3: Not a RSA1 key file /etc/ssh_host_dsa_key.
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: inetd sockets after dupping: 3, 3
debug3: BSM audit: connection from 1.1.1.247 port 53137
debug3: BSM audit: iptype 4 machine ID 010101d9 00000000 00000000 00000000
Connection from 1.1.1.247 port 53137

A successful connection follows...

  1. sshd does no longer start and accept inbound connections in inetd mode (-i), using $ sudo /usr/sbin/sshd -i -d -d -d -e
debug2: load_server_config: filename /etc/sshd_config
debug2: load_server_config: done config len = 493
debug2: parse_server_config: config /etc/sshd_config len 493
debug3: /etc/sshd_config:14 setting Protocol 2
debug3: /etc/sshd_config:30 setting SyslogFacility AUTHPRIV
debug3: /etc/sshd_config:32 setting LogLevel DEBUG3
debug3: /etc/sshd_config:39 setting MaxAuthTries 3
debug3: /etc/sshd_config:108 setting UseDNS no
debug3: /etc/sshd_config:111 setting MaxStartups 5
debug3: /etc/sshd_config:119 setting Subsystem sftp /usr/libexec/sftp-server
debug3: /etc/sshd_config:121 setting IgnoreRhosts yes
debug3: /etc/sshd_config:122 setting IgnoreUserKnownHosts no
debug3: /etc/sshd_config:123 setting PrintMotd yes
debug3: /etc/sshd_config:124 setting StrictModes yes
debug3: /etc/sshd_config:125 setting RSAAuthentication yes
debug3: /etc/sshd_config:126 setting PermitEmptyPasswords no
debug3: /etc/sshd_config:127 setting PasswordAuthentication yes
debug3: /etc/sshd_config:128 setting DenyGroups deniedssh
debug3: /etc/sshd_config:129 setting PubkeyAuthentication yes
debug3: /etc/sshd_config:130 setting GatewayPorts no
debug3: /etc/sshd_config:131 setting AllowTcpForwarding yes
debug3: /etc/sshd_config:132 setting KeepAlive yes
debug1: sshd version OpenSSH_5.2p1
debug3: Not a RSA1 key file /etc/ssh_host_rsa_key.
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug3: Not a RSA1 key file /etc/ssh_host_dsa_key.
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: inetd sockets after dupping: 3, 4
debug3: BSM audit: connection from UNKNOWN port 65535
BSM audit: getaddrinfo failed for UNKNOWN: nodename nor servname provided, or not known
debug3: BSM audit: iptype 0 machine ID 00000000 00000000 00000000 00000000
Connection from UNKNOWN port 65535
SSH-2.0-OpenSSH_5.2

No connection can be established.

Any suggestion in which direction to look for a fix?

Pro Backup
  • 914
  • 4
  • 15
  • 33

1 Answers1

1

It's pretty non-standard to use initd to start anything on a Mac. Instead, launchd is used, kicking off sshd in an ad hoc fashion (ie, it doesn't run as a typical server daemon until there's knock on the door). I suspect that your use of Linux-centric Webmin to manage ssh is contributing to the problem, since Webmin doesn't know a whole lot about launchd.

First, make sure the ssh launchd item is configured to load, just to eliminate the obvious.

sudo launchctl load -w /System/Library/LaunchDaemons/ssh.plist

This is akin to ticking the box on Server Admin.app in the Settings options to enable SSH. Check syslog to see if launchctl is complaining about something.

It's unclear why you would want Webmin to handle SSH, but Apple's default configuration might be illuminating.

There's a launchd item in /System/Library/LaunchDaemons called sshd.plist. This XML file indicates that /usr/libexec/sshd-keygen-wrapper is used as the "program" that actually kicks off /usr/sbin/sshd using the -i flag. (The sshd-keygen-wrapper program is a shell script to first set up initial rsa and dsa keys in empty user home dirs.) The sshd-keygen-wrapper, however, also kicks off sshd like exec /usr/sbin/sshd $@ and is a trusted/whitelisted program as far as the socket firewall is concerned.

You might also want to grab the default /etc/sshd_config from backup or another machine to eliminate that as a variable in troubleshooting.

flumignan
  • 347
  • 2
  • 9
  • My goal is to fix a broken system where I can no longer logon to using ssh. /usr/libexec/sshd-keygen-wrapper /usr/sbin/sshid -i, doesn't that launch sshd in inetd mode? – Pro Backup May 20 '11 at 00:57
  • Reverting to the old version of /etc/sshd_config does not change the situation. – Pro Backup May 20 '11 at 01:41
  • That does launch sshd in inetd mode, but it's launchd that typically kicks it off, and only when someone attempts to connect. Once the session is complete, that sshd process is killed. – flumignan May 20 '11 at 02:34
  • Is the sshd launchd item loading? You can tell like this: `sudo launchctl list | grep sshd` ... by the way, is this Mac OS X Server? Or Mac OS X (desktop client)? – flumignan May 20 '11 at 02:42
  • This is Mac OS X (desktop client): 10.6.5. Output of list is - 0 com.openssh.sshd. Does this mean that sshd launchd is loaded? – Pro Backup May 20 '11 at 09:57
  • No sshd is not loading. The number/pid returned is a 0. On other systems the output is like '80325 - 0x107220.sshd 80320 - 0x104d70.sshd 80319 - com.openssh.sshd '. I was only loading and unloading using 'launchctl load ...' and 'launchtl unload ...'. After adding 'launchctl start com.openssh.sshd', the list command outputs 1 line: '12058 - com.openssh.sshd', but repeating the list command later, returns the 1 liner with a zero again: '0 - com.openssh.sshd'. It turns out that sshd stops in 30 seconds (+/- 4 sec.). No messaging output in console's all messages. – Pro Backup May 20 '11 at 10:11
  • The launchd item _has_ properly loaded. That's what `launchctl list` shows. The dash in the first column simply notes that there is no pid on your local system because the process has not _started_. There's a difference between a launchd item loading and starting. Again, **sshd does not run constantly like it does on other operating systems**. Of course it stops after 30 seconds, because launchd kills it, because there's no external ssh connection coming inbound to maintain. On your other systems, the `launchct list` command shows a PID because you're probably remotely connecting to observe. – flumignan May 20 '11 at 14:03
  • You are right, `launchctl list` on other systems was executed through a remote SSH connection. Good to hear that the launchd sshd item has properly loaded. Understand that sshd does not constantly run, but starts upon demand. Still something is blocking sshd/port 22 because ssh connections return a 'Operation timed out' when sshd is started by launchd, and connections are successful when sshd is started in interactive (not inetd) mode. Firewall/networking issues can be removed from the list. Any suggestions why this ssh connection to sshd returns an 'operation timed out'? – Pro Backup May 20 '11 at 14:17
  • Firewall ruleset? SACLs? Can you try the bonjour/rendezvous/zeroconf/ipv6 name? That's set in System Prefs --> Sharing preference pane on desktop clients, the one with .local domain name. The service access control list is also set in Sharing, under Remote Login --> Allow access for... – flumignan May 20 '11 at 17:18
  • Firewall can be exempted, as starting without detaching is ok and a localhost connection even fails. Bonjour/rendezvous/zeroconf/.local name also fails. SACL: there is one user allowed, and that can connect when sshd is started without detaching. – Pro Backup May 20 '11 at 19:35
  • I have this exact problem. sshd is launched in inetd mode by launchctl. I haven't changed the server side, but two client computers now do this when connecting to the server. I suspect it has something to do with rejecting the RSA1 keys, then seeing a DSA key and DSA no longer being supported? – akohlsmith Jan 16 '17 at 16:56