62

Let's say it again, we all make mistakes, and I have just made one.

A brief history: I was doing some stuff on a VPS (Debian) I'm renting, when I noticed some strange behaviour. Using the netstat command I saw an non-authorized connection through SSH. I didn't know what to do, so I decided to close his connection using iptables:

iptables -A INPUT -p tcp --dport ssh -s IP -j DROP

But I am tired, and I wrote

iptables -A INPUT -p tcp --dport ssh -j DROP

and I kicked myself (and everyone else) out...

How do I fix this?

Aaron Hall
  • 296
  • 3
  • 12
tomatoGuy
  • 621
  • 5
  • 3
  • possible duplicate of [Is it possible to SSH access a server with a misconfigured subnet?](http://serverfault.com/questions/682756/is-it-possible-to-ssh-access-a-server-with-a-misconfigured-subnet) – kasperd Apr 22 '15 at 15:15
  • 1
    no, because he's blocking all ssh packets at the machine. on a misconfigured subnet, you'd just tcpdump for that mac address, get the subnet it's arp-ing on then configure a virtual nic to the same subnet and talk to it. – jfalcon aka Don Fanning Apr 22 '15 at 19:52
  • 51
    Well, you certainly removed access for the unauthorized individual, at least. – user Apr 22 '15 at 22:28
  • @jfalcon They are not exactly the same, but I do believe the same solution applies in both cases. In both cases the misconfiguration is such that ssh over IPv4 is not a feasible option but ssh over IPv6 is unaffected by the mistake. It is possible that one being a virtual server and the other being a physical server has something to say, that certainly looks like a more relevant difference between the two than the exact details of how IPv4 access to the server was broken. – kasperd Apr 23 '15 at 10:54
  • 2
    For next time, perhaps a [dead man's switch](http://serverfault.com/q/232174/31784) would be handy. – Wayne Conrad Apr 23 '15 at 22:48
  • 2
    Who is your host? A lot of VM hosts offer a serial console that lets you SSH in cases where you are unable to from a remote location. – Jon Apr 26 '15 at 18:59

5 Answers5

60

There are several alternatives:

  • See if they have IPMI / "KVM" / console access to the server which lets you control it as if you had a physical keyboard plugged into it.
  • If they don't offer that, see if you can boot the VM to a recovery linux CD (some providers offer this) and then correct the firewall rules that way and then boot it like normal.

  • If you don't have console access, before you boot to recovery or attach the volume to another VM (as in the Amazon case, credit user3550767's answer), you can try Ankh2054's answer of rebooting first if you haven't saved the rules (likely the case since you kicked yourself out before you had a chance to save). Use the control panel or ask someone to power cycle it using a non-graceful reset / poweroff (aka hard reboot or hard shutdown) in case the init script saves the rules automatically when gracefully rebooting (credit @jfalcon, @joshudson).

    Weigh the drawbacks of this (such as data being written during reboot may be lost and filesystem check may be required on boot so longer boot up time, though that delay may be less than booting to recovery).

Braiam
  • 622
  • 4
  • 23
g491
  • 973
  • 5
  • 7
  • 1
    depending on the vps server, there might not be a remote access that he can reach. systems like vmware,kvm,xen,etc.. have consoles but are not setup for public consumption. closes that does allow this form of public control is openstack. – jfalcon aka Don Fanning Apr 22 '15 at 19:49
  • 4
    that said, if it's in amazon/google space, he could snapshot the drive and mount it from another virtual machine to do the fix. – jfalcon aka Don Fanning Apr 22 '15 at 19:50
47

If you have not yet saved the IPtables rule, you can reboot server on VPS (if available) and the rule should disappear.

Ankh2054
  • 1,404
  • 13
  • 23
  • unless the init script saves the iptables during shutdown. – jfalcon aka Don Fanning Apr 22 '15 at 19:46
  • 3
    @jfalcon: That's why you can ask them to pull the virtual plug. – joshudson Apr 22 '15 at 20:49
  • 22
    @jfalcon That's also why it's a bad idea to auto-save on shutdown... make the save a conscious decision by the sysadmin, not something blindly executed by the system. – user Apr 22 '15 at 22:27
  • @joshudson: You have to tell the reboot monkey exactly what to do and force power off. Not just initiate a shutdown. – jfalcon aka Don Fanning Apr 23 '15 at 04:05
  • @MichaelKjörling: This philosophy is wrong as well. This is a VPS so likely he is sandboxing his applications and he has no control over who hits the power switch. And you have no idea how the VPS hosting configured their builds. There is nothing wrong with saving on shutdown. His mistake was his mistake. The smart move in the first place would have been to put SSH on a non-standard port or use fail2ban instead of reacting with a panicked reaction. – jfalcon aka Don Fanning Apr 23 '15 at 04:17
30

This is what human-staffed help lines are for. Call the service provider, and have one of their operators remove the rule for you.

RobertG
  • 301
  • 2
  • 2
3

A general way to fix a broken instance is to shut it down and attach the root volume to a working instance. You can then mount the volume there and view logs or edit configuration files. You can then detach the volume and start it up in its own instance.

user3550767
  • 31
  • 1
  • 2
    True for AWS VPSes; not generally true. – MadHatter Apr 23 '15 at 10:37
  • @MadHatter The exact details may vary, but if a VPS provider doesn't offer any method to boot with a known working image from where you can mount and fix your root file system, then I would expect them to go out of business once customers learned about that limitation. – kasperd Apr 23 '15 at 11:12
  • Usually, they do it by offering console access, so you can boot in single-user mode, or from rescue media (usually abstracted as an ISO). I submit that that is in fact much more normal than AWS's odd way of handling the issue, where a second VPS is (temporarily) a requirement. – MadHatter Apr 23 '15 at 11:16
  • @MadHatter Which of the two approaches seems odd may very well depend on what you are the most familiar with. And either approach gets the job done. Which is the most common I can't really comment on, since so far I have only needed such access on a single VPS provider. – kasperd Apr 23 '15 at 20:11
  • @kasperd: No, which you use depends on which a given VPS provider supports; it's not a matter of choice. If you're using AWS, you mount the volume on another VPS; if you're using a more normal VPS provider, you boot single-user or from rescue media. I don't make this point to be a dick (well, not entirely) but because it's important to appreciate that one has to find out what one's provider's supported method(s) are, then use them. Trying to use the AWS method with, say, Hetzner *simply won't work*, because one Hetzner vserver can't see another's FS (afaik). – MadHatter Apr 23 '15 at 20:29
  • @MadHatter I never said it was a matter of choice. I said the details vary between providers, but in either case you can boot a known working image and mount the problematic image to fix it. In the case of Hetzner you don't get to choose an image, it is the one they constructed for you. I believe they say that image is net booted, though to the user it would look the same if they used a VM feature to load the image directly (such a feature exist in QEMU and possibly other VMs). – kasperd Apr 23 '15 at 20:50
  • some other hosters (at least i know that strato has that) got a recovery system to mount the main sytem offline incase of screwups like this. had a lockout due to a ssh failure, too at my end once but was able to zap the config that way to regain access – masterX244 Apr 24 '15 at 14:49
3

Formal answer: go to VPS management panel, get local access (virtual KVM) somehow or call them.

Explanation of steps/rules to prevent falling for it again:

  1. There are ip, routing and firewall rule changes that may go bad and block your access.
  2. and it applies to dedicated network device configuration too, not to VPSes only

So unless you are 100% sure that you can recover.. I recommend to always make a way to reset network config to its previous state.. like, open background session with either screen, nohup, or tmux, even cron may work for this, and add iptables -F or other desired means to reset anything to the previous state.

kagali-san
  • 1,921
  • 5
  • 17
  • 20