1

I have a bash script which copies small files in succession using scp. So there are a bunch of scp commands in this script. An SSH key is being used to authenticate to the other server.

Because the files are so small, the SSH connection attempts happen very quickly, and not far into the script at all, an scp will hang indefinitely.

  • No errors produced.
  • Network equipment has been eliminated from the equation and the servers communicating live on the same subnet.
  • Issue has been demonstrated with rsync as well.

If a sleep of 1 second is placed between these scp calls, the script runs smoothly, no hang.

Where should I look for this apparent rate limiting? I don't see this between any of my other servers...

anx
  • 6,875
  • 4
  • 22
  • 45
Marcus
  • 175
  • 8
  • 2
    Have you considered using persistent ssh connections? Not only would it solve this problem, it would make the overall transfer much faster. – Michael Hampton Sep 02 '21 at 12:55
  • @MichaelHampton I have considered that actually, but as I am troubleshooting processes which I do not own, I am tasked with fixing the problem without changing the script. Definitely a decent workaround idea though. – Marcus Sep 02 '21 at 13:26
  • @anx This is interesting thanks for the link, I did not know this. However this SSH issue has been demonstrated with rsync as well, so I believe the problem is not with scp. – Marcus Sep 02 '21 at 19:19

1 Answers1

3

Sometimes people setup rate limiting using iptables.

The OpenSSH has the MaxStartups option that does some rate limiting on incoming clients. The default (at least on my computer) is 10:30:100.

man sshd_config

Alternatively, random early drop can be enabled by specifying the three colon separated values start:rate:full (e.g. "10:30:60"). sshd(8) will refuse connection attempts with a probability of rate/100 (30%) if there are currently start (10) unauthenticated connections. The probability increases linearly and all connection attempts are refused if the number of unauthenticated connections reaches full (60).

One other semi-common problem and cause of OpenSSH connection delays is related to an feature on the OpenSSH server that will attempt to do a reverse lookup of the incoming IP address on a connection attempt. I believe it needs this DNS functionality for compatibility with some of the older rhost compatible methods of authentication, functionality that I think almost nobody uses anymore. Anyway the DNS resolution feature will cause problems if the DNS resolvers is badly configured, are configured to use a broken resolver, or perhaps something about the reverse zone the client IP is connecting from is broken.

Ideally the answer is to fix DNS and make sure your DNS is always works without errors, and replies quickly. But if you don't require the DNS resolution, then a quick solution here is to stop the server from attempting to resolve names. Set UseDNS no in your sshd_config.

Zoredache
  • 128,755
  • 40
  • 271
  • 413
  • 1
    Thanks for this response. Unfortunately I believe I have eliminated these, as MaxStartups is set the same as on my hosts which do not exhibit the hang. And iptables contains no rate limiting rules. – Marcus Sep 02 '21 at 19:17
  • 2
    Does it actually hang `indefinitely` or just a long time? One more possibility I can think of is that you have broken DNS on that server, and it is timing out attempting to resolve DNS. You might want to setting `UseDNS no` in your sshd_config. – Zoredache Sep 02 '21 at 22:15
  • 1
    I think you nailed it with UseDNS no. If you'd like to update your answer with this additional tidbit which solved my problem, I'll accept it. – Marcus Sep 03 '21 at 18:49
  • @Marcus what is the background of DNS failing for successive login attempts, thogh? Is this the fault or a broken local stub resolver, or something that really cannot be fixed locally? – anx Sep 03 '21 at 19:25
  • @anx Root cause still under investigation. Knowing it's DNS though it supremely helpful. I had other clues it was DNS but didn't know about SSH's UseDNS until now. – Marcus Sep 04 '21 at 02:21