12

My conclusion to this was to pipe VLAN trunks through EoIP tunnels and encapsulate those in hardware assisted IPSec. Two pairs of fairly inexpensive Mikrotik RB1100AHx2 routers proved capable of saturating a 1 Gbps connection while adding less than 1 ms latency.

I would like to encrypt traffic between two data centres. Communication between the sites is provided as a standard provider bridge (s-vlan/802.1ad), so that our local vlan tags (c-vlan/802.1q) are preserved on the trunk. The communication traverse several layer 2 hops in the provider network.

Border switches on both sides are Catalyst 3750-X with the MACSec service module, but I assume MACSec is out of the question, as I don't see any way to ensure L2 equality between the switches over a trunk, although it may be possible over a provider bridge. MPLS (using EoMPLS) would certainly allow this option, but is not available in this case.

Either way, equipment can always be replaced to accommodate technology and topology choices.

How do I go about finding viable technology options that can provide layer 2 point-to-point encryption over ethernet carrier networks?

edit:

To sum up some of my findings:

  • A number of hardware L2 solutions are available, starting at USD 60,000 (low latency, low overhead, high cost)

  • MACSec may in many cases be tunneled through Q-in-Q or EoIP. Hardware starting at USD 5,000 (low-medium latency, low-medium overhead, low cost)

  • A number of hardware assisted L3 solutions are available, starting at USD 5,000 (High latency, high overhead, low cost)

Roy
  • 4,256
  • 4
  • 35
  • 50
  • 1
    Is there a reason to do it at Layer-2 rather than using IPSec between hosts? – mfinni Jul 01 '13 at 13:16
  • Layer 2 connectivity is a requirement. One would think that encrypting a layer 2 network on layer 2 rather than doing tunneling and fork lifting would be faster, simpler and more secure. However, IPSec/L2TP or similar (with encryption and encapsulation done in ASIC) may still turn out to be the best available option; that is essentially what I'm trying to figure out. – Roy Jul 01 '13 at 15:02
  • I might add that the price tag of two ASAs capable of maintaining 1 Gbps full duplex IPSec adds some motivation for exploring the alternatives. By comparison you can get a Catalyst supporting 10 Gbps/wirespeed MACSec for less. – Roy Jul 01 '13 at 15:09
  • There are a ton of devices that use proprietary ways to do this. I don't think there is a standard or anything. – Falcon Momot Jul 01 '13 at 15:18
  • Have you actually tried this? I don't understand how your provider adding and then removing a tag would mess up macsec. The frame that the far switch receives should be identical to the frame sent. – longneck Jul 01 '13 at 15:22
  • Either the header is encrypted, or changes made to the header as it traverses a non MACSec bridge breaks frame integrity. Putting a dumb switch between the two MACSec switches certainly breaks the protocol. I see the theoretical possibility of sending the encrypted traffic through Q-in-Q provided the frame is dumbly forwarded without interpretation or modification, after all that's why EoMPLS works. I have not been able to confirm whether this could work or not. – Roy Jul 01 '13 at 16:08
  • Have you considered idea's like encapsulation like creating an IPSec tunnel between two devices at each end of the layer 2 link and then running L2TP/L2TPv3 over the tunnel, bridging with the L3 device's LAN interface? – jwbensley Jul 01 '13 at 16:20
  • Yes, I have considered fork lifting to layer 3 and doing IPSec encryption and tunnel through L2TP or GRE ; see my first comment. I'm not sure I see any advantages with such solutions. Does L2TPv3 offer any improvements on latency and small packet overhead? Is there reason to think that IPSec encryption hardware capable of saturating 1 Gbps full duplex might have a lower cost that can make up for the disadvantages of encapsulation and fork lifting? – Roy Jul 02 '13 at 10:24
  • Cisco units now have the interface command "macsec dot1q-in-clear 1" to leave the vlan tag on the outside of the encryption. The problem is this option doesn't seem to exist in 3560 3750 switches. Only seems to be the newer IOS XE. – Brain2000 Oct 27 '17 at 00:18

2 Answers2

5

I just did a quick search for "CESG layer 2 encryption" (CESG are a british government agency who specialise in assurance for computer systems), on Google, and found a few options on their list, there's at least one that'll do 1Gbit, and a few that'll do up to 10Gbit.

It'd probably (almost definitely) be overkill, but you'll find that there's quite a lot of milspec products that are capable of Layer 2 encryption, at quite high throughputs.

The first one I found is VLAN and MPLS agnostic, unsurprisingly, but I suspect they're bloody expensive.

Tom O'Connor
  • 27,440
  • 10
  • 72
  • 148
  • 1
    I don't know about overkill, the CN1000 was already my backup plan, if a less expensive solution can not be found – Roy Jul 01 '13 at 16:28
  • What's pricing on those bad boys like? – Tom O'Connor Jul 01 '13 at 19:09
  • In these parts I believe they are listed around $35,000 (+ tax) per unit (1 Gbps ethernet edition) – Roy Jul 02 '13 at 10:27
  • About as much as I expected, I was pondering if they'd be 100K+ – Tom O'Connor Jul 02 '13 at 10:30
  • Considering you get 20 Gbps worth of MACSec from leading vendors for less than $3,500 I still think the layer 2 devices I know about are insanely overpriced. One might pay 200 times more for the same bandwidth and comparable encryption latency. – Roy Jul 02 '13 at 14:10
  • I'm not sure the L2 encryption capabilities of the equivalent cisco is CESG validated to the extent the CN1000 is.. Depends how secure you need it. – Tom O'Connor Jul 02 '13 at 15:42
  • There are few formal requirements in this case, other than the need for reasonable measures to prevent theft of data in transit. I suppose one could argue that widespread use provides validation. If there's something wrong with AES-GCM I'm sure we'll hear about it. – Roy Jul 02 '13 at 16:34
0

Encryption solutions for Metro/Carrier Ethernet differ quite substantially from MacSec, which was designed for LANs and not for WANs. There is a marketoverview consisting of three documents (intro, P2P, multipoint). Google for "Metro Carrier Ethernet Encryptor" and you'll find it.

Concerning pricing it is imperative to differentiate between list prices and market prices. A 1Gb encryptor will currently cost you around $20K. If you put that into relation with the line costs, it is obvious that encryptor costs are only high if compared to non-comparable solutions.

Christoph Jaggi
  • 1
  • I think part of the point is that WANs and LANs are growing a lot closer together technology wise. About line costs, around these parts, the cost of upgrading from virtual wire to dedicated wire/frequency (where MACSec is obviously fully supported) is a LOT less than acquiring dediacted L2 encryptors. We're talking an order of magnitude. – Roy Jul 30 '13 at 18:46