What's the best fail-safe backup option to access the firewall in case the network becomes unreachable?

Question

We're hosting 12 servers on a rack in a datacenter with minimally sufficient administrative skills to do the job right now.

We have VPN to a firewall and a DMZ and internal network configured.

But what happens the first day that I can't reach the firewall, we get locked out, or a mis-configuration takes down access to the VPN and admin interface?

We have backup options right? 3G, modems? What's the typical, simple, cheap approach to safeguarding against a mis-configuration or network change that causes our network to be unreachable?

The firewall is a SonicWall 3500.

Answer 1

Assuming that a clustered firewall with independent configurations is not possible or desired, I would set up a low-power host to act as a secondary, redundant firewall (but without live traffic).

NOTE: I didn't call this a backdoor or recommend setting up one. A backdoor to a firewall screams danger. A better way to ask this question would be, what's a safe, fallback plan to access a firewall in an event of a misconfiguration.

On that host, I would:

• Connect it to the internet and internal network (or DMZ network), making sure it does NOT go through the main firewall
  • If you have secondary modems with static IP's, then you can use that internet connection instead
• It MUST be configured as secure as (or even more secure than) your main firewall. For example:
  • Use a updated, hardened Linux distribution
  • Have only one listening service: SSH
  • Accept only SSH keypair authentication on a non-root user (sudo if you must later)
  • Limit what networks can access that host (say if your office has a static subnet)
  • You can even get fancy and obfuscate the host even more with things like non-standard port number for SSH or port knocking (probably overkill though, and obfuscation is NOT security)

Once logged in via SSH, you should be able to SSH or SSH tunnel to wherever you need within your internal network.

This is still not 100% fool proof because there are a few rare cases this will fail. For example, if your ISP messes up their own network configuration (unless you have a secondary modem/ISP).

Answer 2

Use a console server and whatever connectivity you can get into the site (3G/DSL/phone line) to connect to it. Some have integrated VPN capability.

There are lots of manufacturers, so you can spend a lot or pick one that fits your budget.

Then even if the firewall is stuck in ROM after a failed upgrade with no network capability, you can still access it remotely.

Answer 3

This is an older solution, but it still works. The methods and protocols are simple and solid.

1. Remote hands at the Datacenter. You'll need to direct a human at the Datacenter to plug in some cables. If there is no remote hands available, you can modify the other items below.
2. A phone (It's useful to have a landline in the datacenter anyways, unless you are okay with the sound quality of a cell phone when surrounded by thousands of computer fans).
3. A 56K modem-- Ensure all cables are ready and are clearly labeled where they go. Assume that the datacenter tech may be distracted, inexperienced or incompetent, or really really tired at 3AM--- hope for the best, plan for the worst. Leave the modem powered off and leave the cables disconnected as an "air gap" defense in case the modem is accidentally powered on. You don't want the modem to be available to in case some wardialer discovers it.
4. A serial console server connected to your most important systems, including your firewall. Some sites use the serial console on a regular basis, so it is well tested.

When you need it, call the datacenter and ask them to plug in the modem and power it on. Then dial in, connect to the firewall and fix the problem.