I am deploying DMVPN. The main complexity is that some spokes are behind NAT and those can't transfer traffic directly to each other, so required to transfer it through hub. However, it dosn't work as expected. The traffic is still being sent directly to other spoke.
The topology is one cloud, dual hub. In the test lab (GNS3) I have four spokes: two of them have "direct" access to spoke (no address translation) and two others are behind other router wihch does NAT, each spoke has its own (so no two spokes behing a single nat).
Dynamic routing is done with EIGRP, because Cisco explicitly recommends it in later DMVPN-related documents, and says OSPF doesn't get excessive testing. However, I am not sure and could use OSPF if it can perform better.
(some hosts are shown as down in the picture, this picture is made after I made all described tests and analysis)
Relevant configs are: hub1:
!
interface Tunnel0
bandwidth 1000000
ip address 10.100.255.253 255.255.255.0
no ip redirects
ip mtu 1400
no ip next-hop-self eigrp 65000
no ip split-horizon eigrp 65000
ip nhrp map multicast dynamic
ip nhrp network-id 65000
ip nhrp holdtime 300
ip nhrp redirect
tunnel source Ethernet0/0
tunnel mode gre multipoint
!
interface Ethernet0/0
description internet.e0/0
ip address 172.31.0.2 255.255.255.0
!
interface Ethernet0/3
description service.e0/0
ip address 10.100.200.253 255.255.255.240
standby 1 ip 10.100.200.252
standby 1 priority 110
standby 1 preempt delay reload 60
standby 1 name HSRP
!
router eigrp 65000
network 10.100.200.176 0.0.0.15
network 10.100.255.0 0.0.0.255
!
ip route 0.0.0.0 0.0.0.0 172.31.0.1
!
hub2:
!
interface Tunnel0
bandwidth 800000
ip address 10.100.255.254 255.255.255.0
no ip redirects
ip mtu 1400
no ip next-hop-self eigrp 65000
no ip split-horizon eigrp 65000
ip nhrp map multicast dynamic
ip nhrp map multicast 172.31.0.2
ip nhrp map 10.100.255.253 172.31.0.2
ip nhrp network-id 65000
ip nhrp holdtime 300
ip nhrp nhs 10.100.255.253
ip nhrp shortcut
ip nhrp redirect
tunnel source Ethernet0/0
tunnel mode gre multipoint
!
interface Ethernet0/0
description internet.e0/1
ip address 172.31.0.3 255.255.255.0
!
interface Ethernet0/3
description service.e0/1
ip address 10.100.200.254 255.255.255.240
standby 1 ip 10.100.200.252
standby 1 preempt delay reload 60
standby 1 name HSRP
!
router eigrp 65000
network 10.100.200.240 0.0.0.15
network 10.100.255.0 0.0.0.255
!
ip route 0.0.0.0 0.0.0.0 172.31.0.1
!
spoke1 (direct internet):
!
interface Loopback0
ip address 10.100.200.5 255.255.255.252
!
interface Tunnel0
bandwidth 100000
ip address 10.100.255.1 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp map 10.100.255.253 172.31.0.2
ip nhrp map 10.100.255.254 172.31.0.3
ip nhrp map multicast 172.31.0.2
ip nhrp map multicast 172.31.0.3
ip nhrp network-id 65000
ip nhrp holdtime 300
ip nhrp nhs 10.100.255.253
ip nhrp nhs 10.100.255.254
ip nhrp shortcut
tunnel source Ethernet0/0
tunnel mode gre multipoint
!
interface Ethernet0/0
description internet.e1/0
ip address 172.31.1.2 255.255.255.0
!
router eigrp 65000
network 10.100.200.4 0.0.0.3
network 10.100.255.0 0.0.0.255
!
ip route 0.0.0.0 0.0.0.0 172.31.1.1
!
spoke2 is similar to 1, only addresses differ. lo0 has 10.100.200.9/30
spoke3 (behind NAT):
!
interface Loopback0
ip address 10.100.200.13 255.255.255.252
!
interface Tunnel0
bandwidth 100000
ip address 10.100.255.3 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp map 10.100.255.253 172.31.0.2
ip nhrp map 10.100.255.254 172.31.0.3
ip nhrp map multicast 172.31.0.3
ip nhrp map multicast 172.31.0.2
ip nhrp network-id 65000
ip nhrp holdtime 300
ip nhrp nhs 10.100.255.253
ip nhrp nhs 10.100.255.254
ip nhrp shortcut
tunnel source Ethernet0/0
tunnel mode gre multipoint
!
interface Ethernet0/0
description spoke3-gw.e0/3
ip address 192.168.1.2 255.255.255.0
!
router eigrp 65000
network 10.100.200.12 0.0.0.3
network 10.100.255.0 0.0.0.255
!
ip route 0.0.0.0 0.0.0.0 192.168.1.1
!
spoke3-gw (the one which does NAT for spoke3):
!
interface Ethernet0/0
description internet.e1/2
ip address 172.31.3.2 255.255.255.0
ip nat outside
ip virtual-reassembly in
!
interface Ethernet0/3
description spoke3.e0/0
ip address 192.168.1.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
!
ip nat inside source list 1 interface Ethernet0/0 overload
ip route 0.0.0.0 0.0.0.0 172.31.3.1
!
access-list 1 permit 192.168.1.0 0.0.0.255
!
spoke4 and spoke4-gw pair is similar, lo0 has 10.100.200.17/30. "grey" addresses are same 192.168.1.0/24. Cisco explicitely states this is valid case and should work.
internet has all those 172.31.{0,1,2,3,4}.1 addresses and just plain forwards packets between .2-s and .0.3; service is just a system which has 10.100.200.241 address and defaults to 10.100.200.252 (HSRP-managed).
There is no IPSEC configured. I am trying to do things gradually, it is overhelming to try to do everyting right from the first shot.
I've done tests with commands like "ping 10.100.200.13 source 10.100.200.17" (from spoke3 to spoke4) and so on.
This "seems" to work. I.e. the said ping works. But capturing packets on link "internet.e1/3 - spoke4-gw.e0/0" showed that we still have direct packets from spoke3-gw to spoke4-gw and vice versa. "show ip nhrp" on spoke3 and spoke4 shows exactly this (resolved to other's -gw "white" address). Inspecting NAT "show ip nat translations" on -gw's showed that yes, there is translation of GRE from spoke3 to spoke4-gw on spoke3-gw (and similar on spoke4-gw). This is the way why said ping works. I am just lucky in this lab to have Cisco doing NAT, which is doing it right.
In the wild world NAT boxes will be anything. I can't require use of some concrete equipment (it is illegal to do so). I only can require it to do NAT of GRE to hubs and back correctly (which is minimum to run DMVPN).
If I block direct traffic between spoke3-gw and spoke4-gw (on internet system, by installing access-rules denying from 172.31.4.2 to 172.31.3.2 and other way around), I expect the system will see that and forward everyting between spoke3 and spoke4 via hub. However, it doesn't work.
sh ip nhrp on spoke3 this time shows it is unable to resolve 10.100.200.17:
10.100.200.12/30 via 10.100.255.3
Tunnel0 created 00:00:39, expire 00:04:33
Type: dynamic, Flags: router unique local
NBMA address: 192.168.1.2
(no-socket)
10.100.200.17/32
Tunnel0 created 00:00:39, expire 00:02:25
Type: incomplete, Flags: negative
Cache hits: 2
10.100.255.3/32 via 10.100.255.3
Tunnel0 created 00:01:34, expire 00:04:20
Type: dynamic, Flags: router unique local
NBMA address: 192.168.1.2
(no-socket)
10.100.255.4/32 via 10.100.255.4
Tunnel0 created 00:00:39, expire 00:04:33
Type: dynamic, Flags: router implicit used nhop
NBMA address: 172.31.4.2
(Claimed NBMA address: 192.168.1.2)
10.100.255.253/32 via 10.100.255.253
Tunnel0 created 03:24:10, never expire
Type: static, Flags: used
NBMA address: 172.31.0.2
10.100.255.254/32 via 10.100.255.254
Tunnel0 created 03:24:10, never expire
Type: static, Flags: used
NBMA address: 172.31.0.3
Why it doesn't direct packets via hub in this case? This way is still open.