2

We have a Bitbucket Server instance hosted in AWS. From some other AWS servers (in another region), git clone over SSH fails with

ssh: connect to host (hostname) port 7999: Connection refused

However, other servers in AWS (in the same region as the Bitbucket server) can successfully clone over SSH, using the same URL.

Other information:

  • Bitbucket is definitely listening on port 7999:

    $ sudo netstat -tnlp | grep :7999

tcp6 0 0 :::7999 :::* LISTEN 20707/java

    (process 20707 is the main Bitbucket process)

  • Bitbucket is running behind Apache as a reverse proxy to provide SSL.

  • tcptraceroute and mtr on port 7999 from the instance that can't clone to Bitbucket on port 7999 successfully connect.
  • The public key for the keypair used by the instance that can't connect is added to the repository's access keys with read access on Bitbucket.
  • In the AWS security groups, the cloning instance's group allows all outbound connections, and the Bitbucket server's groups allow connections from anywhere on port 7999.

  • Results from a tcpdump on port 7999 on the cloning instance:

    tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

16:56:53.348387 IP (tos 0x0, ttl 64, id 61715, offset 0, flags [DF], proto TCP (6), length 60)
(cloning server's hostname).38606 > (bitbucket server's hostname).irdmi2: Flags [S], cksum 0x2799 (incorrect -> 0xb1db), seq 3675985409, win 26883, options [mss 8961,sackOK,TS val 1512892178 ecr 0,nop,wscale 7], length 0

16:56:53.489908 IP (tos 0x0, ttl 252, id 37586, offset 0, flags [none], proto TCP (6), length 40)
(bitbucket server's hostname).irdmi2 > (cloning server's hostname).38606: Flags [R.], cksum 0x24e4 (correct), seq 1472002966, ack 3675985410, win 26883, length 0

  • Results from ssh'ing from the cloning server to Bitbucket:

     $ sudo ssh -vvv -p 7999 ssh://git@stash.tddevops.com
 OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013
 debug1: Reading configuration data /root/.ssh/config
 debug1: Reading configuration data /etc/ssh/ssh_config
 debug1: /etc/ssh/ssh_config line 56: Applying options for *
 debug2: ssh_connect: needpriv 0
 debug1: Connecting to stash.tddevops.com [172.24.16.201] port 7999.
 debug1: connect to address 172.24.16.201 port 7999: Connection refused
 ssh: connect to host stash.tddevops.com port 7999: Connection refused

  • tcpdump on the Bitbucket server shows no data when a clone is attempted.

DylanSp
  • 83
  • 1
  • 8
  • On the 'other AWS server (in another region)', run `tcpdump -n port 7999` and compare traffic caused by `git clone ...` and `telnet 7999` (assuming `telnet` succeeds, if not use `tcptraceroute`). – sborsky Oct 31 '17 at 16:27
  • @sborsky There's a lot more traffic caused by git clone, as one might expect, but I'm not sure what differences I'm looking for. – DylanSp Oct 31 '17 at 16:54
  • You're looking for the TCP request sent by `git`, that results in `Connection refused`. Compare that to a successful one (e.g. sent by `telnet`). The difference should give a clue on what is going wrong. – sborsky Oct 31 '17 at 16:58
  • @sborsky I added the tcpdump output from the cloning instance to the question; does that help? – DylanSp Oct 31 '17 at 17:09
  • Could you post the `tcpdump -n` output for a successful connection, to compare? – sborsky Oct 31 '17 at 18:48
  • Try a `tcpdump` on the BitBucket instance and try to connect. If anything, you should see a `RST` packet sent by the bitbucket server if it's rejecting the connection (you see this on the cloning machine's side as `FLAGS [.R]`, but curious to see if it does it on both). Also, post the bitbucket server's iptables output. – Nathan C Nov 02 '17 at 13:01
  • What does 'ssh -vvv' say? – cptMikky Nov 04 '17 at 23:13
  • @mikky Added to the question. – DylanSp Nov 06 '17 at 15:46
  • @NathanC tcpdump on Bitbucket shows nothing when I try to connect. Iptables on Bitbucket doesn't show any rules, it allows anything. – DylanSp Nov 06 '17 at 15:50
  • Do you have IP overlap in your AWS VPCs? `172.24.16.201` is an internal/local IP address. Try from `other-region` using the bitbuckets' external IP rather than the hostname. – AWippler Nov 06 '17 at 16:32
  • @AWippler Doesn't appear to be the issue; if I curl the Bitbucket server's URL (on port 443, we have HTTPS enabled), I get the web interface. The Bitbucket server isn't externally exposed. – DylanSp Nov 06 '17 at 16:40
  • You say tcpdump on the bitbucket server shows nothing. Thus your problem lies between. Try this: look at the ttl value shown in your tcpdump (for the reset packet). It is 253, presumably it left whatever blocked it at 255, so it seems the blocker is likely 2 hops from you. ie. firewall policies closer to home, probably due to the 172.16.0.0/12 private address you are attempting to connect to. Perhaps in the working case you have something in your local SSH config to override IP for that hostname? – Cameron Kerr Nov 07 '17 at 04:23
  • @CameronKerr You were on the money; it was due to some firewall rules set up by our Security department. – DylanSp Nov 07 '17 at 17:39

2 Answers2

0

From https://serverfault.com/a/288493/442063:

You see the "incorrect" checksums due to a feature called TCP checksum offloading. The checksum fields for outgoing TCP packets are not pre-calculated by the operating system but instead set to 0 and left for calculation by the NIC processor. The Wireshark FAQ has a more detailed explanation.

That might help, but your problem sounds like a firewall issue. It couldn't hurt to make sure you do not have an internal firewall running on this hosted AWS server. Without knowing what OS you are running, I can't really speculate on what to look for. Many Linux operating systems use iptables. For example, you can see here how to turn off and disable the firewall for Oracle Linux or Redhat Linux.

Bill
  • 126
  • 12
  • It's a CentOS server, for the record. I checked with iptables -L; there aren't any rules, iptables allows any connections. – DylanSp Nov 06 '17 at 15:35
0

I found out the issue; it was an external issue with our Security department's firewall. Nothing needed to be changed on the cloning server or the Bitbucket server.

DylanSp
  • 83
  • 1
  • 8