3

What I would like to achieve

I want to securely spread an existing internal subnet over multiple buildings. That means that I have two locations with virtual machines that need to be within the same subnet. The idea is that the virtual machines (having a static IP) can be migrated from one location to the other.

The (physical) host machines are connected to a switch at each location. So, if there wasn't any security or cost problem I would simply connect both switches with a network cable:

[Machines]---[Switch A] <---- LONG CABLE ---> [Switch B]---[Machines]

What I would like, is to replace this long cable by an encrypted tunnel using two gateways that don't need to care about IP adresses or routing and just take any incoming packets encrypt them and send them to the other gateway via an encrypted tunnel. The other gateway then decrypts the packets and sends them to the remote switch. This would physically look like this:

[Machines]---[Switch A]---[GATEWAY A] <-- INTERNET --> [GATEWAY B]--[Switch B]---[Machines]

I would like to avoid that the gateways need any IP adresses within the subnet. The rules shall be completly port-based:

  • Incoming data at port 1: route through tunnel interface
  • Incoming data at tunnel interface: route via port 1

The two Gateways would have a static, routeable IP address to establish the tunnel. The encryption shall be strong (at least AES128, SHA256, DH2048; shared secret is fine), which simple PPP type tunnels don't support. So an additional/seperate encryption layer might be needed.

I've only MikroTik Routers available. So I would prefer to use them. However, I'm mostly looking for the 'magic words' (protocol names and the like) and the right combination of technology that allows me to do that. So, if you know how to do it with Cisco routers or HP routers, it would probably also help, if you explained how you would just do it with that ones...

Questions/Attempts

What kind of firewall filters and protocols can I use to achieve this?

My first idea was to use IPsec to span the encrypted tunnel. But, then I would need to define IPsec Policy that is physical-port based. But there is only an option to define that data from/to a special IP-adress / IP-port combination.

So IPsec would just work as encryption layer for another tunnel type (PPTP, SSTP, L2TP and OVPN are currently supported by the MikroTik RouterOS). As PPP-Tunnels typically don't support strong encryption, I would let IPsec do this job and span the unencrypted-PPP-Tunnel through the encrypted IPsec-tunnel.

Ok, now we had at least some tunnel interface, that we can use like a outgoing port. However, I'm kind of lost here. I don't find that possibility to say: "a frame with incoming at has to be sent out via interface " and "a frame incoming at has to be sent out via interface ".

I'm not often working at Layer2... so I'm actually looking for the right 'term' or 'category'. I could imagine finding it at the IP-Firewall (mangle->prerouting) or something like that, but I assume that's already Layer 3 stuff...

Do I just need to setup a bridge? If so, how can I add the tunnel-interface to the bridge (preferably using the winbox-Interface)? Does the bridge need a MAC adress?

Just in case this attempt is a dead end: I also found "EoIP", "IP Tunnel" and "GRE Tunnel" at the "Interface" setting. But I've no real idea what they can do... So just in case, let me know which of them is worth investigating...

Also, If there is a more easy-and-clean solution don't mind just telling me your solution... You don't have to continue my above attempts, if there is just an easier way!

SDwarfs
  • 385
  • 4
  • 15
  • I retracted my answer because it does not do encryption like you want and was geared towards wireless. – Eddie Dunn Dec 11 '15 at 14:46
  • 2
    Forget about OpenVPN - Mikrotik team just lies about of support of OpenVPN, it's functionally is greatly limited and bugged. If you have "public addresses" on your mikrotiks' interfaces then @Cha0s idea is a way to go, if no then you're tosed... PPTP encryption is too weak, L2TP has no encryption at all and SSTP is compatible with nothing. You could use metarouter with OpenWRT to have full functionality of OpenVPN but it'll kill your tunnel performance. I am afraid that if you need it then IPSec is the only way. PS. AFAIK there's also a possibility to wrap L2TP around IPSec. – Michal Sokolowski Dec 13 '15 at 06:22
  • 1
    If public IPs are not available, there is still a way to implement this but it will kind of suck in terms of performance. You can have EoIP on top of IPsec on top of PPTP (it's ridiculous I know lol). The MTU will be quite low, but it will work. I have this kind of implementation on a site for over 6 years now without a glitch. It's just that it won't perform that good (especially if the uplinks have limited bandwidth) since there will be too much overhead for each packet. – Cha0s Dec 13 '15 at 12:29
  • This is something important form RouterOS 6.33.3 you can use L2TP/EoIP with IPsec key from the box without setup IPsec as is. – kgimpel Jan 13 '16 at 13:03
  • @kgimpel can you share some more information about it, like some example? – Michal Sokolowski Jun 13 '17 at 10:21

1 Answers1

5

You can accomplish what you ask by using a combination of bridges, EoIP tunnels and IPsec.

First you create an EoIP tunnel on both sides so that both routers can talk to each other.
EoIP is a MikroTik proprietary tunnel that works quite like GRE proto 47.
It can forward whole ethernet frames not just IP (like IPIP or TAP interface for example) making it ideal to 'expand' L2 broadcast domains over L3 links.

Then you setup IPsec to encrypt the communications over the tunnel.
On the latest versions of Mikrotik this can be done automatically from the EoIP interface settings, though I prefer setting this up manually for better control.
You apply the IPsec policies on the EoIP tunnels endpoint IPs (so encrypting the whole EoIP tunnel traffic) and not for the networks passing through the tunnel.

And finally you create a bridge on both routers and attach the ethernet port(s) and the EoIP tunnel on each side.

This way you can expand the L2 broadcast domain to the remote side and the bridge will take care of which packets should go over the tunnel and which should stay locally.
Bridges are like software switches so they keep a table of which MACs are on which ports and automatically forward packets that are destined to which ever port. All broadcast traffic will be forwarded to the tunnel of course. More info on bridges

On the official MikroTik documentation this scenario is already documented (with the exception of the IPsec encryption).

Let us assume we want to bridge two networks: 'Office LAN' and 'Remote LAN'. By using EoIP setup can be made so that Office and Remote LANs are in the same Layer2 broadcast domain.

Consider following setup: enter image description here

As you know wireless station cannot be bridged, to overcome this limitation (not involving WDS) we will create EoIP tunnel over the wireless link and bridge it with interfaces connected to local networks.

We will not cover wireless configuration in this example, lets assume that wireless link is already established

At first we create EoIP tunnel on our gateway ... 

[admin@Our_GW] interface eoip> add name="eoip-remote" tunnel-id=0 \
\... remote-address=10.0.0.2
[admin@Our_GW] interface eoip> enable eoip-remote
[admin@Our_GW] interface eoip> print
Flags: X - disabled, R - running
  0    name=eoip-remote mtu=1500 arp=enabled remote-address=10.0.0.2 tunnel-id=0
[admin@Our_GW] interface eoip>

... and on Remote router 

[admin@Remote] interface eoip> add name="eoip" tunnel-id=0 \
\... remote-address=10.0.0.1
[admin@Remote] interface eoip> enable eoip-main
[admin@Remote] interface eoip> print
Flags: X - disabled, R - running
  0   name=eoip mtu=1500 arp=enabled remote-address=10.0.0.1 tunnel-id=0

[admin@Remote] interface eoip>

Next step is to bridge local interfaces with EoIP tunnel On Our GW ...

[admin@Our_GW] interface bridge> add 
[admin@Our_GW] interface bridge> print
Flags: X - disabled, R - running
 0  R name="bridge1" mtu=1500 arp=enabled mac-address=00:00:00:00:00:00 
      protocol-mode=none priority=0x8000 auto-mac=yes 
      admin-mac=00:00:00:00:00:00 max-message-age=20s forward-delay=15s 
      transmit-hold-count=6 ageing-time=5m 
[admin@Our_GW] interface bridge> port add bridge=bridge1 interface=eoip-remote
[admin@Our_GW] interface bridge> port add bridge=bridge1 interface=office-eth
[admin@Our_GW] interface bridge> port print
Flags: X - disabled, I - inactive, D - dynamic
 #    INTERFACE      BRIDGE  PRIORITY PATH-COST
 0    eoip-remote    bridge1 128      10
 1    office-eth     bridge1 128      10
[admin@Our_GW] interface bridge>

... and Remote router: 

[admin@Remote] interface bridge> add 
[admin@Remote] interface bridge> print
Flags: X - disabled, R - running
 0  R name="bridge1" mtu=1500 arp=enabled mac-address=00:00:00:00:00:00 
      protocol-mode=none priority=0x8000 auto-mac=yes 
      admin-mac=00:00:00:00:00:00 max-message-age=20s forward-delay=15s 
      transmit-hold-count=6 ageing-time=5m 
[admin@Remote] interface bridge> port add bridge=bridge1 interface=ether
[admin@Remote] interface bridge> port add bridge=bridge1 interface=eoip-main
[admin@Remote] interface bridge> port print
Flags: X - disabled, I - inactive, D - dynamic
 #    INTERFACE      BRIDGE  PRIORITY PATH-COST
 0    ether          bridge1 128      10
 1    eoip-main      bridge1 128      10     
[admin@Remote] interface bridge>

Now both sites are in the same Layer2 broadcast domain. You can set up IP addresses from the same network on both sites.

And you also need to setup IPSec. More info here

Keep in mind that IPsec while the most secure way to communicate on Mikrotik at this moment it's rather CPU hungry unless your routers have hardware acceleration. So if you are using simple routerboard don't expect anything more than 10-15mbits of throughput.

If your routers have dynamic internet IPs then you will need to use the IP > Cloud feature of MikroTik to get a Dynamic DNS and then use those DynDNS with a little scripting to automatically update the IPs on the EoIP tunnels and IPsec peers and policies when your public IPs change.

Finally I assume that your connections to the internet have at best a MTU of 1500bytes. Using the tunnel and EoIP you will need to lower the EoIP tunnel interface MTU in the realm of 1400bytes (you'd better do some testing to find out the best value for your setup/traffic).
For my setup/traffic I use 1400bytes. Unfortunately due to the nature of IPsec depending on the size of your packets the overhead is variable so you can't always get the best out of it.

Cha0s
  • 2,432
  • 2
  • 15
  • 26
  • Nice answer! Did you compare EoIP with L2TP in the terms of the performance using routerboards? OVPN with OpenWRT in metarouter won't give more than 5 mbps using simple boards like rb951. – Michal Sokolowski Dec 13 '15 at 06:29
  • 2
    L2TP does not forward Ethernet frames so it won't work for this use case. Also OpenVPN on Mikrotik is very bad. It does not support UDP and LZO so it's pretty much useless in terms of performance that's why I didn't suggest this solution. In any case EoIP has very little CPU requirements since it just encapsulates the packets (like GRE does). It doesn't do any encryption on its own (which is what takes up the CPU in OpenVPN for example). – Cha0s Dec 13 '15 at 11:20
  • Thank you for sharing your findings. Especially ` EoIP on top of IPsec on top of PPTP` is nice catch. :) – Michal Sokolowski Dec 13 '15 at 13:36
  • Thank's for you explanation! Looks quite promising. I'll try this setup today and let you know the results. I'm aware that the devices like RB750GS are quite slow on encryption/decryption. For the final setup, I'll use some devices from the CloudCore-Series. One of them also has a dual power supply, which makes the solution also immune against simple power outages. – SDwarfs Dec 14 '15 at 12:31
  • PS: The connection between the routers is a switches network. I assume having 1 GBit/s or even more, that is why I'll definitively need one of the more powerful CPUs with AES support in hardware, to ensure the connection isn't toooo slooooowwww. I'll talk to the network admins about the MTU of the network between the buildings, I assume it's at least 1500 (standard frame sizes). However, thank you for mentioning the issue, so one doesn't forget about it... And other people might want to use DSL-Uplinks, so they will be happy having this information, too. – SDwarfs Dec 14 '15 at 12:39
  • 1
    You could also ask your network admins to increase the MTU between the two buildings (if possible) so that you would end up with 1500bytes MTU over the tunnel without fragmentation. This would ensure maximum performance :) – Cha0s Dec 14 '15 at 13:24
  • 1
    YEAH! It's working... finally! Had problems with the IPSec tunnel, not creating any Security Associations. Found out that I shouldn't touch the default Policy, but create a new Policy for my tunnel. I also used the same IP-Adress for EoIP endpoints and IPSec endpoints, and told the IPSec Policy to only catch protocol 47 (this is what the EoIP-built-in IPSec-Settings do, but they can't be changed and don't have strong encryption configured). Measured speed between a RB750GS and a RB1200 (got 16Mbit/s up, 8Mbit/s down). Now, I only need to switch to faster devices to speed things up. THANKS! – SDwarfs Dec 14 '15 at 18:40
  • I am glad it worked for you :) I haven't used the automatic setup of IPsec via the EoIP interface since this is a recent feature and I've gotten used to setting this manually over the years. It's a bit more work but you have full control of IPsec settings that way. – Cha0s Dec 14 '15 at 19:40
  • Let us know how performance will look like after devices upgrade. :) And share your config for future generations. ;) – Michal Sokolowski Dec 15 '15 at 10:56
  • @StefanK. before going spending money on CCRs check this thread first http://forum.mikrotik.com/viewtopic.php?f=1&t=84465 since it seems that CCRs perform very *VERY* badly with tunnels over IPsec. You may want to look into an x86 solution (which is more work to build than getting a ready-made appliance) to get the performance you need. – Cha0s Dec 18 '15 at 17:06
  • Another alternative maybe using the opensource EoIP tunnel interface for Linux and ditching Mikrotik altogether. https://code.google.com/p/linux-eoip/ – Cha0s Dec 18 '15 at 17:08
  • @Cha0s: Thx, don't know if a x86-Solution is acceptable to my bosses. Anyways, this was a working proof of concept implementation and it worked. Now it seems that another department is in charge of doing the real implementation... bit anoying, but at least I learned something. – SDwarfs Feb 20 '16 at 21:14
  • Regarding CCRs, there has been a recent update to MikroTik which greatly improves the IPsec performance. – Cha0s Apr 13 '17 at 12:27