Site-to-site IPSec routing (Ubuntu, StrongSwan)

Question

I am stuck in trying to connect two networks.

SiteA: is a number of VPS in different locations and office workstations connected with OpenVPN in a private network 10.113.0.0/24. Each has it's own internet access and default gateway. OpenVPN server has public ip 95.95.95.95 and also stands for the IPSec endpoint.

SiteB: is a private network build on VMWare cloud behind the Edge Gateway with public ip 212.212.212.212.

The problem is that hosts in SiteB can't be accessed from hosts in SiteA. Meanwhile they are accessible from the SiteA endpoint directly. I'm sure that I missed something really simple and obvious.

SiteA Endpoint configs

Ifconfig

eth0      Link encap:Ethernet  HWaddr 00:00:00:46:76:84  
      inet addr:95.95.95.95    Bcast:95.95.95.255  Mask:255.255.255.0
      UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
      RX packets:249492177 errors:0 dropped:395296 overruns:0 frame:0
      TX packets:13730009 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1000 
      RX bytes:39576084672 (39.5 GB)  TX bytes:8388420192 (8.3 GB)

tun0      Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
      inet addr:10.113.0.1  P-t-P:10.113.0.1  Mask:255.255.255.0
      UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
      RX packets:13968 errors:0 dropped:0 overruns:0 frame:0
      TX packets:11554 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:100 
      RX bytes:1069136 (1.0 MB)  TX bytes:4425784 (4.4 MB)

IPSec Connection:

conn portal-vcd
    authby=secret
    auto=start
    ## phase 1 ##
    ike=aes128-md5-modp1024    
    ikelifetime=28800           
    keyexchange=ikev1          
    ## phase 2 ##
    esp=aes128-sha1-modp2048
    type=tunnel
    leftid=95.95.95.95
    left=95.95.95.95
    leftsubnet=10.113.0.0/24
    rightid=212.212.212.212
    right=212.212.212.212
    rightsubnet=10.200.0.0/24

Traceroute from Endpoint is OK

traceroute 10.200.0.1
traceroute to 10.200.0.1 (10.200.0.1), 30 hops max, 60 byte packets
1  10.200.0.254 (10.200.0.254)  3.042 ms  2.979 ms  2.943 ms
2  10.200.0.1 (10.200.0.1)  3.457 ms  3.472 ms  3.051 ms

IP xfrm policy:

ip xfrm policy
src 10.113.0.0/24 dst 10.200.0.0/24 
    dir out priority 2883 
    tmpl src 95.95.95.95 dst 212.212.212.212
        proto esp reqid 1 mode tunnel

Routes table:

route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         95.95.95.254    0.0.0.0         UG    0      0        0 eth0
10.113.0.0      *               255.255.255.0   U     0      0        0 tun0
95.95.95.0      *               255.255.255.0   U     0      0        0 eth0

IPTABLES NAT:

Chain POSTROUTING (policy ACCEPT 174 packets, 12723 bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1       96  5928 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            policy match dir out pol ipsec   
2        0     0 MASQUERADE  all  --  *      eth0    10.113.0.0/24        0.0.0.0/0

SiteA VPS Config

Ifconfig:

eth0      Link encap:Ethernet  HWaddr 00:00:00:34:22:23  
      inet addr:178.X.X.X  Bcast:178.X.X.255  Mask:255.255.254.0
      inet6 addr: fe80::200:ff:fe34:2223/64 Scope:Link
      UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
      RX packets:5599082 errors:0 dropped:48196 overruns:0 frame:0
      TX packets:300211 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1000 
      RX bytes:918538724 (918.5 MB)  TX bytes:706426556 (706.4 MB)

tun0      Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
      inet addr:10.113.0.2  P-t-P:10.113.0.2  Mask:255.255.255.0
      UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
      RX packets:384 errors:0 dropped:0 overruns:0 frame:0
      TX packets:1590 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:100 
      RX bytes:80459 (80.4 KB)  TX bytes:124217 (124.2 KB)

I added a static route

route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         178.X.X.X       0.0.0.0         UG    0      0        0 eth0
10.113.0.0      *               255.255.255.0   U     0      0        0 tun0
10.200.0.0      10.113.0.1      255.255.255.0   UG    0      0        0 tun0
178.X.X.0    *                  255.255.254.0   U     0      0        0 eth0

Traceroute shows that packet is sent to 10.113.0.1 but as seems to me not routed to IPSec there.

traceroute 10.200.0.1
traceroute to 10.200.0.1 (10.200.0.1), 30 hops max, 60 byte packets
 1  10.113.0.1 (10.113.0.1)  1.887 ms  2.453 ms  2.452 ms
 2  * * *

What is misconfigured? Thanks!!

Edited after @MadHatter comments:

Tcpdump of ICMP request #ServerA tun0:

tcpdump -n -n -i tun0 host 10.200.0.1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type RAW (Raw IP), capture size 262144 bytes
08:52:32.011631 IP 10.113.0.2 > 10.200.0.1: ICMP echo request, id 24258, seq 10, length 64
08:52:33.011614 IP 10.113.0.2 > 10.200.0.1: ICMP echo request, id 24258, seq 11, length 64

Tcpdump of ACMP response on ServerA after traffic returned form tunnel:

tcpdump -n -n -i eth0 host 10.200.0.1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
08:52:45.014288 IP 10.200.0.1 > 10.113.0.2: ICMP echo reply, id 24258, seq 23, length 64
08:52:46.014010 IP 10.200.0.1 > 10.113.0.2: ICMP echo reply, id 24258, seq 24, length 64

List of all iptables rule. As @MadHatter said in comments forwarding must be on by default. But I mentioned that it is for tun0 -> eth0 traffic, but for eth0 -> tun0 it doesn't.

iptables -L -n -v
Chain INPUT (policy ACCEPT 3380K packets, 1704M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy DROP 1095 packets, 91824 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 2886  236K DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
 2886  236K DOCKER-ISOLATION  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
 0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
 0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
 0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
 0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           
 0     0 ACCEPT     all  --  eth1   eth0    0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
 1020 85632 ACCEPT     all  --  tun0   eth0    0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
 637 47172 ACCEPT     all  --  tun0   eth0    0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 105K packets, 13M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER-ISOLATION (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 2886  236K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
24177   70M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

I added the forward rule for eth0 -> tun0

iptables -A FORWARD -i eth0 -o tun0 -j ACCEPT

And it's magigally worked! HostA:

ping 10.200.0.1    
PING 10.200.0.1 (10.200.0.1) 56(84) bytes of data.
64 bytes from 10.200.0.1: icmp_seq=1 ttl=62 time=3.17 ms
64 bytes from 10.200.0.1: icmp_seq=2 ttl=62 time=2.88 ms

+1 in Karma for the @MadHatter.

Firstly, when a site A client is trying to address a host in Site B (say, `ping 10.200.0.1`), do you see this traffic coming *out* of site A's OpenVPN server tunnel interface (`Aserver# tcpdump -n -n -i tun0 host 10.200.0.1`)? — MadHatter, Dec 31 '17 at 10:31
@MadHatter yes. I see traffic: tcpdump -n -n -i tun0 host 10.200.0.1 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on tun0, link-type RAW (Raw IP), capture size 262144 bytes 17:17:20.397205 IP 10.113.0.2 > 10.200.0.1: ICMP echo request, id 14708, seq 1, length 64 17:17:21.396362 IP 10.113.0.2 > 10.200.0.1: ICMP echo request, id 14708, seq 2, length 64 — Ivan Yaremchuk, Dec 31 '17 at 14:17
OK, now check whether it's coming out of the ipsec tunnel, with eg `serverB# tcpdump -n -n -i eth1 host 10.200.0.1` (assuming `eth1` is the internal interface). Basically, check at each step along the chain, to see if you can identify where it's getting lost. It's also generally better for you to edit these outputs into your question, as it preserves formatting and keeps the information in one place. — MadHatter, Dec 31 '17 at 15:50
@MadHatter thank you for giving the methology of network debug. Now it seems clear and easy :) I think that I found the problem. The traffic goes to tunnel and returned back but SideA server don't redirect it back to the HostA. I can see response ICMP packets on eth0 ServerA. Edited the original question with dumps. — Ivan Yaremchuk, Jan 03 '18 at 05:59
How can I tell Ubuntu to redirect reply packets to 10.113.0.0/24 network? — Ivan Yaremchuk, Jan 03 '18 at 06:07
It ought to do it by default, especially given that it was happy enough to forward the traffic outbound. Could we see the **whole** of serverA's firewall, with `iptables -L -n -v; iptables -t nat -L -n -v`? — MadHatter, Jan 03 '18 at 07:38
@MadHatter I fixed it :) Deep gratitude to the community that helps. I am a fullstack developer not an admin and all these things are a little bit of magic for me because of lack of knowledge in proper debug tools and rules. Edited the original post for newcomers can understand what is going on. I cant raise your carma because of comments but I did it in my mind)) — Ivan Yaremchuk, Jan 03 '18 at 09:28
No worries, we're glad to help - all the community asks of you is that when you have an answer that works for you, you accept it. I'm going to write up what we did as an answer now, so come back in an hour and see if you like it. For my part, I thank you for a good, clear first question - the diagram was a *big* help. — MadHatter, Jan 03 '18 at 09:31
It's also much easier to help someone who - as you do - seizes on the clues provided and goes off to follow them on his/her own initiative, instead of waiting to be led slowly through each step of the proceedings. One makes ones own luck, largely! — MadHatter, Jan 03 '18 at 09:48

score 8 · Accepted Answer · answered Jan 03 '18 at 09:39

We have used tcpdump to examine traffic in and out of the two firewall nodes. I note in passing that tcpdump with {Open,Libre,Strong}S/WAN in a modern kernel can be a bit problematic, because on the interface out of which the encrypted traffic comes and goes one sees the plaintext traffic only when it leaves and not when it arrives.

Nevertheless, using tcpdump to follow the flow, we have established that the ICMP echo-requests are getting all the way from network A to network B, and responses are getting as far back as serverA (the network A OpenVPN server / IPSec tunnel collapse point), but they are not passing through it to the OpenVPN client.

Since the traffic is being forwarded outbound, there is no general problem with traffic forwarding, and thus we suspect firewall rules. You have added a rule to permit the forwarding of traffic from the external network to the OpenVPN tun0 interface, and complete connectivity has resulted.

You might wish to refine that rule slightly, eg to have it explicitly apply to traffic that arrived via an IPSec connection

iptables -A FORWARD -i eth0 -o tun0 -m policy --pol ipsec --dir in -j ACCEPT

or perhaps to make it statefully-aware, but those are refinements and are up to you.

Site-to-site IPSec routing (Ubuntu, StrongSwan)

1 Answers1