18

We have a server where one of our engineers misconfigured the subnet and now we are locked out of this server and the only access that I know of would work is a serial console from IDC (it means asking an IDC engineer to help us with this).

What has been misconfigured:

address 192.168.1.9 # Original address there was
netmask 255.255.255.254 # Misconfigured, originally should've been .240

Out of curiosity - is there a way to avoid calling IDC and somehow connect to this host over SSH (then we can fix the configuration)?

Peter Mortensen
  • 2,319
  • 5
  • 23
  • 24
Alexey Kamenskiy
  • 774
  • 9
  • 22

3 Answers3

25

You need to be able to log in on another host on the same network segment. Some of the ways to get access to the misconfigured host requires root on the intermediate host, but there also is one easy way to get access without needing root on the intermediate host.

The easy way to access the host using IPv6

ssh -o ProxyCommand='ssh -W [fe80::42:ff:fe:42%%eth0]:%p user@intermediate-host' root@target-server

The following example values in above command need to be substituted with correct values for your use case: fe80::42:ff:fe:42, eth0, user, intermediate-host, and target-server.

Detailed explanation of how it works

ProxyCommand is an ssh feature to use when you cannot open a TCP connection directly to the target host. The argument to ProxyCommand is a command whose stdin/stdout to use instead of a TCP connection.

-W is used to open a single port forwarding and connect it to stdin/stdout. This fits nicely together with ProxyCommand.

fe80::42:ff:fe:42%%eth0 is the link-local address of the target host. Notice that due to ProxyCommand using % as escape character, the typed ssh command must use %% in that location. You can find all link-local addresses on the segment by running ssh user@intermediate-host ping6 -nc2 ff02::1%eth0.

Using IPv6 link-local addresses for this purpose is usually the easiest way because it is enabled by default on all modern systems, and link-local addresses keep working even if both IPv4 and IPv6 stacks are severely misconfigured.

Falling back to IPv4

If IPv6 is completely disabled on the misconfigured host (absolutely not recommended), then you may have to resort to using IPv4. Since IPv4 doesn't have link-local addresses the way IPv6 does then accessing the misconfigured host using IPv4 gets more complicated and need root access on the intermediate host.

If the misconfigured host was still able to use its default gateway, you would be able to access it from outside. Possibly the misconfigured netmask also broke the default gateway due to the stack refusing to use a gateway outside of the prefix covered by the netmask. If this is indeed the case, the misconfigured host will only be able to communicate with 192.168.1.8 because that's the only other IP address in the subnet currently accessible to this misconfigured host.

If you have a login on 192.168.1.8, you might just be able to ssh from there to 192.168.1.9. If 192.168.1.8 is currently unassigned you can temporarily assign it to any host on the segment on which you have root access.

kasperd
  • 29,894
  • 16
  • 72
  • 122
  • And `fe80::42:ff:fe:42` is the address of...? my misconfigured server I guess? – Alexey Kamenskiy Apr 15 '15 at 06:43
  • @AlexKey Yes, it needs to be substituted with the link-local IPv6 address of the misconfigured server. – kasperd Apr 15 '15 at 06:48
  • 1
    Any example with accessing via IPv4 (in case of IPv6 is disabled)? – Alexey Kamenskiy Apr 15 '15 at 06:56
  • @AlexKey In case IPv6 is disabled I think you have to go through 192.168.1.8 because it appears to be the only other IP beneath the configured prefix. – kasperd Apr 15 '15 at 07:01
  • @AlexKey I assume you are only asking about the case of IPv6 being disabled out of academic interest. I mean you wouldn't be so careless as to completely disable IPv6 on a production server. – kasperd Apr 15 '15 at 07:03
  • Thanks a lot! With slight changes I managed to salvage the server! I will accept your answer as it gives great deal of details (was enough for me as crash course) and same time post my own answer with detailed flow of how I did it. – Alexey Kamenskiy Apr 15 '15 at 07:27
  • @kasperd why shouldn't you disable IPv6 on a production server if it's not being used? – Lenniey Apr 15 '15 at 07:43
  • 1
    @Lenniey apparently for reason like this question (at least). – Alexey Kamenskiy Apr 15 '15 at 07:44
  • @AlexKey well...enabling a potential security risk (what IPv6 is if not maintained properly) to "protect" from misconfiguration is not what I have in mind for production servers – Lenniey Apr 15 '15 at 07:46
  • @Lenniey There are systems where disabling IPv6 is not an officially supported configuration. Additionally if you have any servers which are not already running native dual stack, you should be planning how to get there. Six years ago there was a significant number of client with a broken IPv6 setup which would make them unable to access dual stack servers. That is no longer the case. The number of users with native IPv6 and no public IPv4 address is increasing. Those will get more reliable connectivity over IPv6 than over IPv4. – kasperd Apr 15 '15 at 08:04
  • @Lenniey Even if you are not ready to create AAAA records for your server, there is little reason to completely disable IPv6. Being able to reach a server using link-local addresses is just one of the things which is easier with IPv6 than with IPv4. Redundancy is very useful in creating reliable systems. And dual stack does provide a kind of redundancy we haven't seen before. We may as well take advantage of that. (Before IPv4 gets turned off, I am sure somebody will figure out how to achieve this kind of redundancy using two IPv6 connections.) – kasperd Apr 15 '15 at 08:09
9

kasperd posted a great answer detailed enough for me to learn how I can recover from situation in the question. This answer is the exact step-by-step of how I did it.

  1. SSH to server on same physical network
  2. Using arp -a or ip neighbor list as root find the MAC address of misconfigured server.
  3. Using MAC to link-local converter find link-local for misconfigured server
  4. Now can SSH to server as any user via ssh user@link-local%dev where:
    • user - username for which we are allowed to SSH
    • link-local - self-assigned IPv6 address recovered in step 3
    • dev is physical interface this server is reachable from (e.g. eth0)
josh3736
  • 249
  • 1
  • 3
  • 11
Alexey Kamenskiy
  • 774
  • 9
  • 22
2

You need to load an IP address within the configured subnet of your target, not just within the range that you actually want.

If you send a packet to 10.0.0.2 with subnet 255.255.255.248 from, for example, 10.0.0.220, 10.0.0.2 will look at its subnet mask to figure out how to reply. Since .220 is WAAY out of the subnet 255.255.255.248, .2 has to send the reply to the default gateway instead.

So if you can load an IP address within the same subnet as .2, eg. 10.0.0.3, then it will work.

In your specific case, for 10.0.0.9,the subnet 255.255.255.254 only has 1 additional IP address, namely 10.0.0.8. So if you can load that IP address, you should be able to SSH.

Peter Mortensen
  • 2,319
  • 5
  • 23
  • 24
thelogix
  • 389
  • 1
  • 7