I seem to have encountered a strange problem with configuring OpenVPN on an AWS EC2 instance (Ubuntu AMI). Connection gets established, two-factor authentication passes, but once connected, the client can't access the Internet.
For debugging, I started an endless ping on the client:
$ ping 87.250.250.242
PING 87.250.250.242 (87.250.250.242): 56 data bytes
64 bytes from 87.250.250.242: icmp_seq=0 ttl=54 time=33.589 ms
64 bytes from 87.250.250.242: icmp_seq=1 ttl=54 time=31.275 ms
64 bytes from 87.250.250.242: icmp_seq=2 ttl=54 time=42.907 ms
64 bytes from 87.250.250.242: icmp_seq=3 ttl=54 time=49.470 ms
64 bytes from 87.250.250.242: icmp_seq=4 ttl=54 time=29.772 ms
On the server I'm testing how far does the packet go. It does reach tun0 interface:
$ sudo tcpdump -i tun0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type RAW (Raw IP), capture size 262144 bytes
08:52:10.261730 IP ip-10-8-1-6.ec2.internal > ya.ru: ICMP echo request, id 52451, seq 2684, length 64
08:52:11.264706 IP ip-10-8-1-6.ec2.internal > ya.ru: ICMP echo request, id 52451, seq 2685, length 64
08:52:12.268201 IP ip-10-8-1-6.ec2.internal > ya.ru: ICMP echo request, id 52451, seq 2686, length 64
08:52:13.272732 IP ip-10-8-1-6.ec2.internal > ya.ru: ICMP echo request, id 52451, seq 2687, length 64
08:52:14.275066 IP ip-10-8-1-6.ec2.internal > ya.ru: ICMP echo request, id 52451, seq 2688, length 64
However, it does not go any further - eth0 doesn't see it:
$ sudo tcpdump -i eth0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
Internet from inside the EC2 instance is perfectly reachable:
$ curl google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>
Looks like a typical NAT postrouting misconfiguration, but I triple-checked that everything is set up as needed.
IP forwarding is on:
$ cat /proc/sys/net/ipv4/ip_forward
1
Postrouting rule in iptables is present:
$ sudo iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DOCKER all -- anywhere anywhere ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
DOCKER all -- anywhere !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- ip-172-17-0-0.ec2.internal/16 anywhere
MASQUERADE all -- ip-10-8-1-0.ec2.internal/24 anywhere
Chain DOCKER (2 references)
target prot opt source destination
RETURN all -- anywhere anywhere
Explicit check also says it is there:
$ sudo iptables -t nat -C POSTROUTING -s 10.8.1.0/24 -o eth0 -j MASQUERADE && echo ok
ok
Ubuntu has ufw installed, but it is not active:
$ sudo ufw status
Status: inactive
Here is more information on the system:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.5 LTS
Release: 16.04
Codename: xenial
$ openvpn --version
OpenVPN 2.4.4 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Sep 26 2017
library versions: OpenSSL 1.0.2g 1 Mar 2016, LZO 2.08
$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.2.50.1 0.0.0.0 UG 0 0 0 eth0
10.2.50.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.8.1.0 10.8.1.2 255.255.255.0 UG 0 0 0 tun0
10.8.1.2 0.0.0.0 255.255.255.255 UH 0 0 0 tun0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
Server config:
$ cat /etc/openvpn/server.conf | egrep -v '^#' | egrep -v '^$'
port 1194
proto udp
dev tun
ca ca.crt
cert VPN.crt
key VPN.key # This file should be kept secret
dh dh2048.pem
server 10.8.1.0 255.255.255.0
ifconfig-pool-persist ipp.txt
push "redirect-gateway def1 bypass-dhcp"
push "dhcp-option DNS 8.8.8.8"
keepalive 10 120
tls-auth ta.key 0 # This file is secret
compress lzo
user nobody
group nogroup
persist-key
persist-tun
status openvpn-status.log
verb 4
plugin openvpn-plugin-auth-pam.so openvpn
reneg-sec 43200
reneg-bytes 0
cipher AES-256-CBC
ncp-ciphers AES-256-GCM:AES-256-CBC:AES-128-GCM:AES-128-CBC:BF-CBC
crl-verify crl.pem
Client config:
client
dev tun
proto udp
remote <server IP> 1194
resolv-retry infinite
nobind
persist-key
persist-tun
remote-cert-tls server
key-direction 1
compress lzo
verb 3
auth-user-pass
auth-nocache
reneg-sec 43200
cipher AES-256-CBC
<ca>
-----BEGIN CERTIFICATE-----
# redacted
-----END CERTIFICATE-----
</cert>
<key>
-----BEGIN PRIVATE KEY-----
# redacted
-----END PRIVATE KEY-----
</key>
<tls-auth>
-----BEGIN OpenVPN Static key V1-----
# redacted
-----END OpenVPN Static key V1-----
</tls-auth>
The system information: EC2 instance built from ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20180814 image.
Security groups allow all outbound traffic and the following inbound traffic:
- UDP/1194 from anywhere
- TCP/22 from my location (for debugging)
I read about disabling Source/Destination Check. Not sure it applies to my case, as I don't need to provide access to AWS VPC network, but tried that as well, and it didn't help.
What puzzles me the most, is that we already have a working VPN server set up in a similar way in another region, but for some reason I can't get a new instance to work. So I'm not even sure this is AWS-specific.
Can anyone suggest where to look for the problem?
