scp "lost connection" but ssh works fine

10

1

A server I can ssh into fine has begun refusing to scp.

$ scp ~/tmp/foo user@some.example.com:~/tmp/
lost connection

With scp -v -v I can see the connection succeeds and the transfer appears to succeed, but no file appears on the other side.

OpenSSH_5.9p1, OpenSSL 0.9.8r 8 Feb 2011
debug1: Reading configuration data /Users/schwern/.ssh/config
debug1: /Users/schwern/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 20: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to testcurrent01.dev.liquidweb.com [10.30.152.254] port 22.
debug1: Connection established.
debug1: identity file /Users/schwern/.ssh/id_rsa type -1
debug1: identity file /Users/schwern/.ssh/id_rsa-cert type -1
debug1: identity file /Users/schwern/.ssh/id_dsa type -1
debug1: identity file /Users/schwern/.ssh/id_dsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3
debug1: match: OpenSSH_4.3 pat OpenSSH_4*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.9
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
...lots of authentication details...
debug1: Enabling compression at level 6.
debug1: Authentication succeeded (publickey).
Authenticated to user@some.example.com ([1.2.3.4]:22).
debug2: fd 5 setting O_NONBLOCK
debug2: fd 6 setting O_NONBLOCK
debug1: channel 0: new [client-session]
debug2: channel 0: send open
debug1: Entering interactive session.
debug2: callback start
debug2: client_session2_setup: id 0
debug2: fd 3 setting TCP_NODELAY
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 0: request env confirm 0
debug1: Sending command: scp -v -t -- ~/tmp/
debug2: channel 0: request exec confirm 1
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel 0: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 0
debug2: exec request accepted on channel 0
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain -> closed
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug2: channel 0: rcvd close
debug2: channel 0: close_read
debug2: channel 0: input open -> closed
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
Transferred: sent 4576, received 2520 bytes, in 0.0 seconds
Bytes per second: sent 167737.0, received 92372.6
debug1: Exit status 0
debug1: compress outgoing: raw data 135, compressed 121, factor 0.90
debug1: compress incoming: raw data 66, compressed 52, factor 0.79
lost connection

It is a CentOS 5.9 machine.

Things I've checked...

  • I have permission to write to that directory.
  • The user has a sensible shell (/bin/bash).
  • I tried moving my ~/.ssh/config out of the way.
  • scp'ing to that machine from others with entirely different operating systems also fail.
  • The disk is not full.
  • Restarting sshd.

/var/log/secure contains...

Apr  4 14:23:22 some sshd[12576]: Postponed publickey for user from 1.2.3.4 port 33581 ssh2
Apr  4 14:23:22 some sshd[12575]: Accepted publickey for user from 1.2.3.4 port 33581 ssh2
Apr  4 14:23:22 some sshd[12575]: pam_unix(sshd:session): session opened for user user by (uid=0)
Apr  4 14:23:22 some sshd[12575]: pam_unix(sshd:session): session closed for user user

What might I check next?

Schwern

Posted 2013-04-04T14:13:32.130

Reputation: 1 978

2

Not the error I would expect, but just in case, do your ~/.bashrc or ~/.profile or /etc/bash.bashrc or /etc/profile print anything to STDOUT? https://bugzilla.redhat.com/show_bug.cgi?id=20527. And I assume you are using Linux?

– terdon – 2013-04-04T14:16:13.853

Nope. I just get the usual Last login: Thu Apr 4 10:15:28 2013 from 1.2.3.4. – Schwern – 2013-04-04T14:18:38.587

Anything in any of the system logs on the target host? – Flup – 2013-04-04T14:21:33.760

@Flup Looks normal. I posted what shows up in the logs when I connect. – Schwern – 2013-04-04T18:25:30.100

Could you start strace -f -o /tmp/sshd.strace -p [pid of sshd] on the server, try again, then post anything from that file that looks relevant? – Flup – 2013-04-05T07:06:49.680

@Flup Tried that. Nothing appears to be obviously on fire but I'm no strace expert. The only time the target file is mentioned is 3399 execve("/bin/bash", ["bash", "-c", "scp -t -- ~/tmp/"], [/* 8 vars */]) = 0 – Schwern – 2013-04-05T13:23:31.547

Any chance you could post the whole trace somewhere? Make sure to use the -f option so we can see what the subprocess does. – Flup – 2013-04-05T13:24:41.243

@Flup http://schwern.net/tmp/sshd.strace Thanks for looking into it.

– Schwern – 2013-04-05T19:06:00.267

Answers

1

Had the same issue.

If you did a minimal install of Centos, it only install the openssh and openssh-server packages but not the openssh-clients. sudo yum install openssh-clients will fix your issue.

qux

Posted 2013-04-04T14:13:32.130

Reputation: 126

I don't have access to that machine anymore, but that seems a likely answer. – Schwern – 2014-11-13T21:26:46.017

4

scp works by making an ssh connection to the remote host, then launching another copy of the scp program on that host. The two scp instances communicate through the ssh connection to perform the file transfer.

"lost connection" is printed by the local scp program when the ssh connection drops prematurely. The usual reason for that is the scp program on the remote host either failed to start or else it exited prematurely. This could have happened because the scp program doesn't exist on the remote host, or it's not in your command PATH, or it's not marked executable, or it crashed after starting, or something along those lines.

Kenster

Posted 2013-04-04T14:13:32.130

Reputation: 5 474

0

We recently had this issue on one of our systems.

We could appropriately ssh onto the host server, but discovered we could not ssh from the server back onto the machine. This is a good place to investigate, if you cannot do this then you will not be able to use SCP.

In our case, somehow (perhaps a botched install) had replaced our ssh binary files with 0 byte empty files. Whenever "ssh" was executed, nothing happened.

By reinstalling openssh-clients, we rectified the binaries and scp started to work.

yum reinstall openssh-clients

Ultimation

Posted 2013-04-04T14:13:32.130

Reputation: 1

Asked: 2013-04-04T14:13:32.130

Viewed: 39 791 times

Active: 2018-03-13T13:22:40.363