I recently got set up with Comcast Business Class internet. Previously, I was using DSL with another provider and had a single static IPv4 address (VyOS). Now I want to get Comcast set up.
I purchased a static IP address delegation from Comcast. I have five static IPv4 addresses (96.x.x.168/29, of which 168 and 175 are reserved and 174 is the gateway, so I can only use 169-173, or five addresses) and a static /56 of IPv6 addresses (2603:xxxx:xxxx:8700::/56 with a static gateway of 2603:xxxx:xxxx:8700:7454:7dff:feb1:d391 [link local fe80::7454:7dff:feb1:d391], so I should have full use of 2603:xxxx:xxxx:8700::1 through 2603:xxxx:xxxx:87ff:ffff:ffff:ffff:ffff). You'll notice that the static gateway is within the /56 and, indeed, is within the first /64 prefix of the /56 ... I don't know whether that is important or causing me any problems. IMPORTANT NOTE: Since this is Comcast, and they can't do anything simply, we are not allowed to put the modem/router in "bridge mode." Static IP addresses can only work if the router is in normal mode (but you can turn all of the unneeded DHCP, firewall, WiFi, etc. off, which I did). If I put the modem/router in "bridge mode," it forces me into a single DHCP address, and that is as expected per Comcast documentation. The Comcast equipment is a Cisco type-BWG model-DPC3939B Business IP Gateway. It's readouts show the IPv4 and IPv6 static delegations.
I'm using VyOS as the routing software on my router/firewall (which is plugged in to one of the LAN ports on the BIP Gateway). It's an OSS fork of Vyatta, parallel to Brocade or Ubiquiti.
I had no problem using my static IPv4 addresses. I set up 169 as the outgoing interface for all of my private traffic (no destination NAT, just source nat using 169 as the outdoing address). I set up 170-173 using 1:1 NAT for various public-facing servers behind my firewall. It all works perfectly.
IPv6 has been a disaster. I can't get much to work. If I do the following (pretty simple), I can ping Google (2607:f8b0:4002:c07::66) from VyOS:
# set interfaces ethernet eth1 2603:xxxx:xxxx:8700::1/60
# set protocols static route6 ::/0 next-hop 2603:xxxx:xxxx:8700:7454:7dff:feb1:d391
# commit
This results in:
$ ip -6 route get 2607:f8b0:4002:c07::66
2607:f8b0:4002:c07::66 from :: via 2603:xxxx:xxxx:8700:7454:7dff:feb1:d391 dev
eth1 src 2603:xxxx:xxxx:8700::1 metric 0
cache
$ ping6 -c 1 2607:f8b0:4002:c07::66
PING 2607:f8b0:4002:c07::66(2607:f8b0:4002:c07::66) 56 data bytes
64 bytes from 2607:f8b0:4002:c07::66: icmp_seq=1 ttl=46 time=30.6 ms
I can also ping 2603:xxxx:xxxx:8700::1 from a remote server I have access to with known working IPv6. So that's nice, but that is literally the extent of what I can get to work. I can't get IPv6 onto any other machine on the network. I tried interface addresses ending in /56, /60, and /64 with no better results in any of the following scenarios:
As a first example, the address 2603:xxxx:xxxx:8700::1/60 should put everything 2603:xxxx:xxxx:8700::1 through 2603:xxxx:xxxx:870f:ffff:ffff:ffff:ffff in scope of my interface. The default route falls within that range. However, I can only ping Google (and can only ping the address from my remote server) if my address is 2603:xxxx:xxxx:8700::anything/60. 2603:xxxx:xxxx:8700::2/60 works, 2603:xxxx:xxxx:8700::feed/60 works, 2603:xxxx:xxxx:8700:1::1/60 works ... but 2603:xxxx:xxxx:8701::1/60 doesn't work. As soon as I try that, ip -6 route and ping6 tell me that the gateway is "unreachable" and that Google is "unreachable," even though all of those addresses (including the last one) are within the same subnet and as each other and the gateway.
As a second example, I tried setting two addresses on the interface:
# set interfaces ethernet eth1 2603:xxxx:xxxx:8700::1/60
# set interfaces ethernet eth1 2603:xxxx:xxxx:8701::1/60
# commit
This yielded interesting results. ip -6 route says I have a route out through 2603:xxxx:xxxx:8701::1 via 2603:xxxx:xxxx:8700:7454:7dff:feb1:d391 and ping6 no longer says "unreachable," but I never get ping responses. It was easy to determine why. I can ping 2603:xxxx:xxxx:8700::1 from my remote server, but not 2603:xxxx:xxxx:8701::1. Since the route from VyOS to Google is out through 2603:xxxx:xxxx:8701::1, the responses can't route back. Weirdly, if I delete both addresses, commit, and then add them back in reverse order, it magically works:
# set interfaces ethernet eth1 2603:xxxx:xxxx:8701::1/60
# set interfaces ethernet eth1 2603:xxxx:xxxx:8700::1/60
# commit
Now ip -6 route says I have a route out through 2603:xxxx:xxxx:8700::1 via 2603:xxxx:xxxx:8700:7454:7dff:feb1:d391 and, since my pings go out through 2603:xxxx:xxxx:8700::1, ping6 to Google results in successful responses. Great. But 2603:xxxx:xxxx:8701::1 still can't be pinged from the outside world, so I still haven't succeeded in using all of my addresses.
As a third example, since I can at least ping Google from VyOS, I want to assign some static addresses to my internal, publicly-facing servers. I start by adding the address 2603:xxxx:xxxx:8700:92::1/64 to bond0.900, the interface facing my servers (this prefix should cover everything from 2603:xxxx:xxxx:8700:: to 2603:xxxx:xxxx:8700:ffff:ffff:ffff:ffff). Then I add 2603:xxxx:xxxx:8700:92::173/64 to the interface on one of my servers and set its default gateway to 2603:xxxx:xxxx:8700:92::1. Now, from my server, I can ping 2603:xxxx:xxxx:8700:92::1. Great! But I can't even ping 2603:xxxx:xxxx:8700::1, which is just one step away on the router, and I for sure can't ping Google. If I use monitor interfaces, I can see that the pings to Google are at least making it out of eth1, and I'm just not getting any responses. No surprise there, I suppose, given my troubles above. But I don't even see the pings to 2603:xxxx:xxxx:8700::1 getting to VyOS. But that's not really how I intended to use it, I just thought I had to since I couldn't get anything beyond :8700 to work.
So I tried setting eth1 to 2603:xxxx:xxxx:8700::1/64 (still works for pinging Google from VyOS), setting bond0.900 to 2603:xxxx:xxxx:8792::1/64, and setting the server to 2603:xxxx:xxxx:8792::170/64. Now, the server can ping 2603:xxxx:xxxx:8792::1 AND 2603:xxxx:xxxx:8700::1! This is an improvement! However, I can't ping the gateway (2603:xxxx:xxxx:8700:7454:7dff:feb1:d391) or Google from the server. monitor interfaces shows the traffic going out, but nothing comes back in.
And this is just the beginning of what I need to do. I eventually want to carve out two /64 prefixes for stateless configuration on two private LANs/WLANs, but I'm not even getting to that yet. One thing at a time, I can't even get static IPv6 fully working, or working at all past the first /64 of the static /56 delegated to me. Clearly I'm doing something wrong here, but this also smells bad. Why is this so hard?
