Multiple firewalls are a good idea in several cases.
Internet connection shared between office-use and webapp
Less common in these cloudy days, but take the case of a webapp being hosted out of a private datacenter, sited in the company office. If one of the workstations in the company office gets wormed (to use an ancient example) by Slammer, all of that traffic will be egressing through the shared internet connection.
If there is a single firewall protecting both the profit-making webapp and the company "browse over lunch" network, such flagrant network abuse as Slammer can cause a DoS on the firewall. This creates a network-down event for the for-profit web-site.
If two firewalls are used, one around the webapp zone and a second one for corporate internet stuff, the office firewall would DoS itself while the webapp merrily continues earning money.
Secondly, such a firewall can provide protection from internal threats; though that could be done with a sufficiently advanced monolithic firewall as well.
Benefits: Fundamental fault isolation between zones
Multi-homed network: Internet + private network
In this case there are more than one connections, an Internet connection and a private network. The private network may be a physical layer connection, or it could be a VPN tunnel to something like an Amazon VPC.
It is entirely possible that the private network connection, whatever it is, can't terminate at the one firewall a company has. For instance, Amazon VPC connections have some specific requirements that may trigger a hardware purchase. Or maybe the one firewall can only have one external connection and adding a second slightly-more-trusted external connection isn't in its feature-set. In these cases, an additional firewall can help terminate and protect the added network connection.
Benefits: Provides capability not present in original device, physical network isolation between trust zones.
Multi-homed network: multiple Internet connections
This is the same setup as the first case, but the Office internet has it's own connection to the world.
In this case it is entirely possible that the Office Internet connection terminates in the downtown office, where the webapp connection terminates in the rented datacenter out in the suburbs. The physical separation makes two devices an easy choice. There may be a physical connection between the datacenter and the office, or there may be a site-to-site VPN.
In either case, multiple firewalls make a lot of sense.
Additionally, if the office ISP is something slow like Comcast Business while the datacenter ISP is something really fast with an SLA, the datacenter connection will need beefier firewalls to keep up with all that traffic.
Benefits: Enhanced network isolation between zones, ability to terminate site-to-site VPN connections between datacenter and office
Anycast to a single site
This is a rather specialist case, but the only way I can conceive of to do the case you're thinking of. In this case, anycast is used to set up multiple points on the Internet connected with very fast network connections to a central site. The higher level protocol is TCP based.
In this case, firewalls would exist at the perimeter of each anycast entry point to the company network. The multi-path case would require:
- The end point to change network location enough to cause it to change anycast destination (pop switch).
- The end point to not change it's IP address.
In this case traffic would enter on one leg and likely leave down the same leg, or exit wherever the routing tables say to send the replies. After the pop switch traffic would come in through a different leg, but replies would likely stay the same (unless routing tables have changed).
The tricky part here is that the firewall in front of the second leg would not see the connection setup so it wouldn't know about it. If it's a statefull firewall, chances are very high it'll reject it out of hand. A stateless firewall would pass the traffic though.
GEO-IP
In this case multiple sites are run using GeoIP to bring services closer to clients.
To get the case you're thinking of:
- An end-user has to change geographic location enough that the GeoIP service returns a different IP.
- The end-user creates a new TCP connection for each transaction of the higher level protocol.
This is the easiest case. It's just new TCP connections, and the firewalls in front of each GeoIP site would treat each connection like a new connection and allow it. The presumably same server providing services would see each as a new TCP connection and handle it appropriately. In a setup like this, Source NAT would be used on the GeoIP site firewalls to ensure replies are sent down the right leg. No muss, no fuss.
Whether or not the higher level protocol can handle such connections is up to the higher level protocol. That isn't a firewall issue at all. Unless, of course, the firewall is stateful in that particular protocol, at which point a setup like this would fail whenever a flap like this happens.
