connecting remote site via fiber: layer 2 vlans or layer 3 routing?

Question

Greets all,

Back in the old days, when you had two geographically separated sites, links were pretty constricted so we put routers on them and, well, 'routed' between ip subnets-per-site. That was best practice at the time.

Now we have a fiber bundle between the two geographically separated sites. It's our own 'owned' fiber so a middleman isn't a concern. The testing indicates that the bundle can handle multi-gigabyte traffic without a problem. Additionally, the fiber ring has included multiple redundancies including separate physical pathings. All well and good.

Given this, is it still considered 'best practice' to use routing and different subnets between the remote sites? Or can we extend our 'local' (main site) network out to the remote site along with the main site vlans? Is that still considered a suboptimal or even bad practice? More to the point, is there any reason not to? (Aside, I understand the 'backhoe interrupt' issue; the separate physical pathings is expected to handle that contingency).

Other thoughts?

thanks!

Answer 1

Now we have a fiber bundle between the two geographically separated sites. It's our own 'owned' fiber so a middleman isn't a concern... Additionally, the fiber ring has included multiple redundancies including separate physical pathings. All well and good.

Given this, is it still considered 'best practice' to use routing and different subnets between the remote sites? Or can we extend our 'local' (main site) network out to the remote site along with the main site vlans? Is that still considered a suboptimal or even bad practice? More to the point, is there any reason not to? (Aside, I understand the 'backhoe interrupt' issue; the separate physical pathings is expected to handle that contingency).

First, there is no such thing as a best practice in this situation. Big-picture design details such as layer2 / layer3 site interconnections are driven by business needs, budget, the capabilities of your staff, your preferences, and your vendor's feature sets.

Even with all the love for moving VM instances between data-centers (which is much easier with Layer2 interconnects between data centers), I personally still try connect buildings at layer3, because layer3 links generally mean:

1. Lower opex and lower time to problem resolution. The vast majority of network troubleshooting diagnostics are based on IP services. For example, mtr only has layer3 visibility. Thus, layer3 hops are much easier to fix when you're finding packet drops, either due to congestion or errors on the links. Layer3 is also easier to diagnose when you're dealing with multipath issues (compared for instance with non-layer3 multipath such as LACP). Finally, it's way easier to find where a server or PC is when you you can traceroute straight to the edge switch.

2. Smaller broadcast / flooding domains. If you have mismatched ARP / CAM timers, you are vulnerable to unknown unicast flooding. The fix for this is well-known, but most networks I see never bother matching the ARP and CAM timers correctly. End result? More traffic bursts and floods within the layer2 domain... and if you're flooding through your inter-building layer2 links, you're flooding natural network congestion points.

3. Easier to deploy firewalls / ACLs / QoS... all these things can work at layer2, but they tend to work better at layer3 (because vendors / standards bodies have spent at least 15 of the previous 20 years building vendor feature sets preferring layer3).

4. Less spanning-tree. MSTP / RSTP have made spanning-tree much more tolerable, but all flavors of STP still boil down to that nasty protocol which loves to flood broadcasts the wrong direction when you drop a BPDU on an STP-blocking link. When might that happen? Heavy congestion, flaky transceivers, links that go unidirectional (for whatever reason, including human), or links that are running with errors on them.

Does this mean it's bad to deploy layer2 between buildings? Not at all... it really depends on your situation / budget / staff preferences. However, I would go with layer3 links unless there is a compelling reason otherwise.¹ Those reasons might include religious preferences within your staff / mgmt, lower familiarity with layer3 configs, etc...

---

¹For anyone wondering how I handle layer2 data center interconnections when there are layer3 links between the datacenters, I prefer EoMPLS pseudowires if there is no Nexus gear. Theoretically OTV seems like a candidate if I had Nexus, but I personally haven't been there yet. Bottom line, there are solutions for tunneling Layer2 through Layer3 when you have to.

Answer 2

This is kind of a tough one as there are advantages and disadvantages to both approaches. In my previous life where my job duties involved much more networking administration instead of system administration we had maybe two dozen sites within a 12 mile wide geographical area. About half of these sites where configured as separate Layer-3 sites that were routed back to the main office and the other half were configured as "Layer-2" sites (i.e., we just extended the VLAN to that site).

The advantages of the "Layer-2" sites were that they were much simpler to setup and maintain; no routers needed, no updating our static routes, no DHCP-relay, no separate VLAN configuration and so on. The main disadvantages I experienced were non-technical, things like it's much harder to locate a rogue DHCP server when your broadcast domain is in 12 different buildings each a few miles apart. Lots of administrative tasks become trickier when you lack the network compartmentalization of different sites, things like different firewall rules for Office A and Office B but not Office C are difficult when they all share the same VLAN/Subnet. I suppose you could also run up to an issue with broadcasts depending on how many devices you have but with switching technology today being what it is, you can cram a surprising number of hosts into a broadcast domain before it really becomes an issue.

The advantages of the "Layer-3" sites is pretty much inverse of "Layer-2" sites. You get compartmentalization, you can write per-site firewall rules, and you know which particular building that damn Linksys Router is in. The disadvantages are obviously the equipment required to do the routing and the necessary configuration and maintenance. Dynamic routing protocols and things like VTP (if you dare use it!) can ease the configuration burden if your network is suitably complex.

My non-answer answer: Don't compartmentalize unnecessarily (that is, resist the temptation to be overly clever) but don't let the short term easy solution win out where it makes more sense to have separate VLANs/Subnets. As someone who has chased down my share of Linksys Rogue DHCP Servers... er "Routers"... I think there is a strong case for a one VLAN/Subnet per building network design simply to limit the damage these misconfigurations can do. On the other hand if you only have two sites and they're right next door maybe it does make sense for them to share the same VLAN/Subnet.

Answer 3

As many have said, there are good and less good sides to both L2 and the L3-solution. I've been employed by a telephone company earlier and I've also been helping smaller networks get started.

L2-solutions are easier to understand and cheaper if everything works. The works part is usually ruined by someone reconnecting a cable that they thought had been accidentally disconnected. Loop protection and Spanning Tree might be of use but will most likely cause more harm than be of use.

L3-solutions, in my experience, have been harder to understand by the parties I've helped. Cost may also become an issue if the hardware and software has to be supported by a manufacturer. Linux on a x86 machine is a very cost effective and feature filled router.

The benefits of a L3-solution is that loops and other broadcasts are contained within a much smaller domain. The best example is that if someone accidentally creates a loop in one of multiple routed branch offices, only that office disappears while others can continue working.

I'd vote for a L3-routed solution, mostly because of the smaller broadcast domains but also because traffic can be prioritized and firewalled with ease. If someone needs L2-connections they can tunnel them over the routed network and even encrypt the traffic on their own if they wish.

Answer 4

I would recommend layer3 switches which will route at lan speed. If you have good fiber you could run a gigabit network over your fiber with such devices and still benefit from advantage of a routed network (reduced broadcast domain, access lists, etc).

Answer 5

I think, routing is a best choice. Your whole network will crash if fiber became broken. And routing with layer 3 switches (layer 3 switching) is fast as layer 2 switching if you use CEF or something like this.