Cisco BGP Unequal Cost Load Balancing

Question

I'm trying to implement BGP Unequal Cost Load Balancing feature in my network. According to cisco manuals (long: http://www.cisco.com/c/en/us/td/docs/ios/12_2s/feature/guide/fsbgplb.html, short: https://ccieblog.co.uk/bgp/bgp-unequal-load-cost-sharing) I have built such net topology:

R1 - router where I'm trying to implement load balancing for outgoing traffic. VRF table with name nat is used.

R2-R4 - NAT servers running quagga, with default route to R5 shared with R1 over eBGP.

R1 configuration

R1 IOS version: 12.2(33)SXJ4 (s72033-adventerprisek9_wan-mz.122-33.SXJ4.bin)

R2 configuration (R3 R4 only router-id and vlan differs)

In result I have 3 different default routes on R1 with same share count - 1/1 (1:1:1). But proportion 1:2:3 expexted:

R1# sh ip bgp vpnv4 vrf nat 0.0.0.0

Paths: (6 available, best #5, table nat)
Multipath: eiBGP
  Advertised to update-groups:
     2         
  65000
    10.30.227.227 from 10.30.227.227 (10.30.227.227)
      Origin IGP, localpref 100, valid, external, multipath
      Extended Community: RT:192.168.33.4:13
      DMZ-Link Bw 250 kbytes
  65000, (received-only)
    10.30.227.227 from 10.30.227.227 (10.30.227.227)
      Origin IGP, localpref 100, valid, external
      DMZ-Link Bw 250 kbytes
  65000
    10.30.228.228 from 10.30.228.228 (10.30.228.228)
      Origin IGP, localpref 100, valid, external, multipath
      Extended Community: RT:192.168.33.4:13
      DMZ-Link Bw 375 kbytes
  65000, (received-only)
    10.30.228.228 from 10.30.228.228 (10.30.228.228)
      Origin IGP, localpref 100, valid, external
      DMZ-Link Bw 375 kbytes
  65000
    10.30.225.225 from 10.30.225.225 (10.30.225.225)
      Origin IGP, localpref 100, valid, external, multipath, best
      Extended Community: RT:192.168.33.4:13
      DMZ-Link Bw 125 kbytes
  65000, (received-only)
    10.30.225.225 from 10.30.225.225 (10.30.225.225)
      Origin IGP, localpref 100, valid, external
      DMZ-Link Bw 125 kbytes

R1# sh ip cef vrf nat 0.0.0.0/0 internal

0.0.0.0/0, epoch 3, flags rib only nolabel, rib defined all labels, RIB[B], refcount 7, per-destination sharing
  sources: RIB, D/N, DRH
  feature space:
   NetFlow: Origin AS 0, Peer AS 0, Mask Bits 0
   Broker: linked
   IPRM: 0x00018000
  subblocks:
   DefNet source: 0.0.0.0/0
  ifnums:
   Vlan3225(231): 10.30.225.225
   Vlan3227(232): 10.30.227.227
   Vlan3228(233): 10.30.228.228
  path 541B7858, path list 53E3E0D8, share 1/1, type recursive nexthop, for IPv4, flags resolved
  recursive via 10.30.225.225[IPv4:nat], fib 5496C804, 1 terminal fib
    path 541B7BF8, path list 53E3E170, share 1/1, type adjacency prefix, for IPv4
    attached to Vlan3225, adjacency IP adj out of Vlan3225, addr 10.30.225.225 513F6B60
  path 541B78CC, path list 53E3E0D8, share 1/1, type recursive nexthop, for IPv4, flags resolved
  recursive via 10.30.227.227[IPv4:nat], fib 54969B7C, 1 terminal fib
    path 541B7B10, path list 53E3E08C, share 1/1, type adjacency prefix, for IPv4
    attached to Vlan3227, adjacency IP adj out of Vlan3227, addr 10.30.227.227 513F66E0
  path 541B7DC8, path list 53E3E0D8, share 1/1, type recursive nexthop, for IPv4, flags resolved
  recursive via 10.30.228.228[IPv4:nat], fib 54970EAC, 1 terminal fib
    path 541B79B4, path list 53E3E040, share 1/1, type adjacency prefix, for IPv4
    attached to Vlan3228, adjacency IP adj out of Vlan3228, addr 10.30.228.228 513F6560
  output chain:
    loadinfo 51283B80, per-session, 3 choices, flags 0003, 5 locks
    flags: Per-session, for-rx-IPv4
    15 hash buckets
      < 0 > IP adj out of Vlan3225, addr 10.30.225.225 513F6B60
      < 1 > IP adj out of Vlan3227, addr 10.30.227.227 513F66E0
      < 2 > IP adj out of Vlan3228, addr 10.30.228.228 513F6560
      < 3 > IP adj out of Vlan3225, addr 10.30.225.225 513F6B60
      < 4 > IP adj out of Vlan3227, addr 10.30.227.227 513F66E0
      < 5 > IP adj out of Vlan3228, addr 10.30.228.228 513F6560
      < 6 > IP adj out of Vlan3225, addr 10.30.225.225 513F6B60
      < 7 > IP adj out of Vlan3227, addr 10.30.227.227 513F66E0
      < 8 > IP adj out of Vlan3228, addr 10.30.228.228 513F6560
      < 9 > IP adj out of Vlan3225, addr 10.30.225.225 513F6B60
      <10 > IP adj out of Vlan3227, addr 10.30.227.227 513F66E0
      <11 > IP adj out of Vlan3228, addr 10.30.228.228 513F6560
      <12 > IP adj out of Vlan3225, addr 10.30.225.225 513F6B60
      <13 > IP adj out of Vlan3227, addr 10.30.227.227 513F66E0
      <14 > IP adj out of Vlan3228, addr 10.30.228.228 513F6560
    Subblocks:
     None

What am I doing wrong? According to manuals, different dmzlink bw values should cause different load sharing proportion, but in fact - it does not!

UPDATE 1 -- requested by user bangal

R1# show ip bgp all summary

For address family: IPv4 Unicast
BGP router identifier X.X.X.129, local AS number 41096
BGP table version is 22283352, main routing table version 22283352
34749 network entries using 4065633 bytes of memory
61661 path entries using 3206372 bytes of memory
8119/5337 BGP path/bestpath attribute entries using 1299040 bytes of memory
3752 BGP AS-PATH entries using 155474 bytes of memory
2990 BGP community entries using 138266 bytes of memory
146 BGP extended community entries using 5168 bytes of memory
53 BGP route-map cache entries using 1696 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 8871649 total bytes of memory
BGP activity 4716897/4682147 prefixes, 11331539/11269872 paths, scan interval 60 secs

# Here are bgp neighbours from global routing table. Not relevant to the question. IP addresses are hidden 

Neighbor     V       AS    MsgRcvd   MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
X.X.X.1      4       XX219    791704  760380 22283352    0    0 6d17h           1
X.X.X.33     4       XX219 112902498 1315655 22283352    0    0 6d17h           0
X.X.X.238    4       XX772    801422  762830 22283352    0    0 2w5d            0
X.X.X.206    4       XX540   2886112 1313917 22283352    0    0 4w4d         9641
X.X.X.70     4       XX772 188343075 1313853 22283352    0    0 6d14h       25881
X.X.X.78     4       XX772 148265282  941127 22283352    0    0 2w6d        26098

# Here are neighbours for vrf nat.

For address family: VPNv4 Unicast
BGP router identifier X.X.X.129, local AS number 41096
BGP table version is 824, main routing table version 824
1 network entries using 137 bytes of memory
6 path entries using 408 bytes of memory
1 multipath network entries and 3 multipath paths
8119/1 BGP path/bestpath attribute entries using 1299040 bytes of memory
3752 BGP AS-PATH entries using 155474 bytes of memory
2990 BGP community entries using 138266 bytes of memory
146 BGP extended community entries using 5168 bytes of memory
53 BGP route-map cache entries using 1696 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 1600189 total bytes of memory
3 received paths for inbound soft reconfiguration
BGP activity 4716897/4682147 prefixes, 11331539/11269872 paths, scan interval 15 secs

Neighbor        V          AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.30.225.225   4       65000   11003   11443      824    0    0 3d18h           1
10.30.227.227   4       65000    9853   10293      824    0    0 3d18h           1
10.30.228.228   4       65000   10992   11432      824    0    0 3d18h           1

R1# sh ip route vrf nat

Routing Table: nat
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is 10.30.228.228 to network 0.0.0.0

     10.0.0.0/24 is subnetted, 4 subnets
C       10.30.0.0 is directly connected, Vlan30
C       10.30.228.0 is directly connected, Vlan3228
C       10.30.227.0 is directly connected, Vlan3227
C       10.30.225.0 is directly connected, Vlan3225
B*   0.0.0.0/0 [20/0] via 10.30.228.228, 3d18h
               [20/0] via 10.30.227.227, 3d18h
               [20/0] via 10.30.225.225, 3d18h

R1# sh ip bgp vpnv4 vrf nat neighbors

R1 sh ip bgp neighbours output

R1# sh run

R1 running config sensitive information is masked

May be you can post the full router config of R1 for proper investigation and also output of `sh ip route`, `sh ip bgp neighbors`, `sh ip bgp summary`. Please also make sure that the basic networking is functional, that end points are properly connected and configured and reachable to each others (ping) even without a bgp. — Diamond, Nov 15 '15 at 21:56
@bangal. I added information that you've requested in comment. Thanks for attetnion! All nodes in topology are fully functional, ping OK. Even load balance is present. Only proportion is wrong: 1:1:1 instead of 1:2:3. — Shamanu4, Nov 16 '15 at 08:17
Can you please use `maximum-paths 3` instead of `maximum-paths 5` and see the outcome? — Diamond, Nov 17 '15 at 21:45
@bangal. I had set `maximum-paths 3` and it did not solve the issue. — Shamanu4, Nov 18 '15 at 07:14
Does your running config of R1 misses `bandwidth 50000` for 'interface Vlan3228'? Could you please also attach output of 'sh ip bgp '? — Andrey Sapegin, Nov 18 '15 at 09:01
Also in the running-config, as far as I understood, the bandwidthes should be 50kbs-100kbs-150kbs, and your output of `R1# sh ip bgp vpnv4 vrf nat 0.0.0.0` shows 125kbs-250kbs-350kbs... — Andrey Sapegin, Nov 18 '15 at 09:14
@AndreySapegin. Updated R1 config with actual information. Bandwidth was set on vlan3228 but somehow I've missed it in config. See updates [at github commit](https://github.com/shamanu4/stackexchange/commit/308b1de6a572fb8566ee1139adfff362a4b76482). I was trying different combinations of bandwidths and this caused some mess in config dump. Now there is actual information — Shamanu4, Nov 18 '15 at 09:15
@AndreySapegin `sh ip bgp vpnv4 vrf nat 10.30.228.228` displays the same as `sh ip bgp vpnv4 vrf nat 0.0.0.0` . Have you missed `neighbors` keyword? If so, output of `sh ip bgp vpnv4 vrf nat neighbors 10.30.228.228` you can see at the end of [R1 sh ip bgp neighbours output](https://raw.githubusercontent.com/shamanu4/stackexchange/master/Q1/R1_show_ip_bgp_neighbours.txt) — Shamanu4, Nov 18 '15 at 09:22
The guide you mentioned (http://www.cisco.com/c/en/us/td/docs/ios/12_2s/feature/guide/fsbgplb.html) differentiates between `neighbor dmzlink-bw`, which only enables advertisement of bandwidth to neighbours (presented in your config) and `bgp dmzlink-bw`, which enables proportional load balancing (and it seems to be MISSING in your config). Could you try to put `bgp dmzlink-bw` into your running config? — Andrey Sapegin, Nov 18 '15 at 09:26
@AndreySapegin. Fixed it [github commit](https://github.com/shamanu4/stackexchange/commit/e10b388b1b3e9a7ba9cb5c3c4d8d6bc9368ba109). But issue still present. Sorry about mistakes in posted full config. Merely I've been trying different combinations to investigate problem, but with no luck. — Shamanu4, Nov 18 '15 at 09:44
OK, can you please add output of `show ip route `? Hopefully, it should show something like `traffic share count is X` for each route... — Andrey Sapegin, Nov 18 '15 at 10:57
Please also try `maximum-paths eibgp` instead of `maximum-paths`, since you use eiBGP, maybe it will work: `router bgp 100`, `no maximum-paths 3`, `maximum-paths eibgp 3`. — Andrey Sapegin, Nov 18 '15 at 11:00
@AndreySapegin possibly you meant `sh ip route vrf nat 0.0.0.0` ? I see 3 active routes via R2-R4. traffic share count is 1 for each. Also i have changed maximum paths option to eibgp: [github commit](https://github.com/shamanu4/stackexchange/commit/54cf1d7d878a84faf2f3c3c19a1efc677eb9aa9e). No luck - issue not solved. — Shamanu4, Nov 18 '15 at 11:31
Yes, the traffic share count should be different for your routes. One more tip, please check if you are consistent with `bandwidth inherit` command for Port-channel interfaces in your config. — Andrey Sapegin, Nov 18 '15 at 11:32
Are you sure that `maximum-paths eibgp 3` should be configured under `address-family ipv4 vrf nat` and not under `router bgp 100`? Please try in both places and all options (eibgp, ibgp, just maximum-paths) — Andrey Sapegin, Nov 18 '15 at 11:35
@AndreySapegin. Yes I'm sure that `maximum paths` should be under `address-family`. I need multipath feature in vrf instance but not in global routing table. If I put `maximum paths` under `router bgp 100` I get in result only one route via R5 and no routes via R3 and R2. Same result for `ibgp` under address-family, because all neighbours are external. Bandwidth inherit on Port-channel is consistent in config. I removed this line with no effect. [commit](https://github.com/shamanu4/stackexchange/commit/0537475c218cb9e318e4c64b0c226544cb810fc9) — Shamanu4, Nov 18 '15 at 11:42
Ha, `bgp dmzlink-bw` should be also under `address-family` (according to the guide). Please do not forget to try maximum paths with/without eibgp. Could you check with `sh ip route vrf nat 0.0.0.0` to see if traffic share counts are finally changed and write back again? — Andrey Sapegin, Nov 18 '15 at 12:10
@AndreySapegin This may be solution of the issue, but in current IOS version I can't place `bgp dmzlink-bw` under address family. It silently fails and places this option in global router bgp configuration. I'll upgrade IOS this night to 15x branch and see if this behavior will change. Please post an answer to make me possible to upvote/grant bounty. Thanks. — Shamanu4, Nov 18 '15 at 13:14

Andrey Sapegin · Answer 1 · 2015-11-18T14:41:13.120

The key problem seems to be a missing bgp dmzlink-bw option under the address-family in the configuration. Let me, however, summarise my comments here:

bgp dmzlink-bw under address-family. neighbor dmzlink-bw only enables advertisement of bandwidth to neighbours, whilebgp dmzlink-bw enables proportional load balancing itself.
Running-config had bandwidth 50000 option missing for 'interface Vlan3228'
As mentioned in this configuration example, option maximum-paths eibgp 3 could be needed instead of maximum-paths 3
In addition to sh ip bgp vpnv4 vrf nat 0.0.0.0 and other commands mentioned in the original guides (see the question), by Shamanu4 and bangal, it is useful to check if traffic share counts are different for links being load balanced using sh ip route vrf nat 0.0.0.0
Check if there are no other options that could interfere with configuration of load balancing (e.g., bandwidth inherit on Port-channel)

As a general advice, sometimes it is very hard to identify the issue, when you have a large running-config with a lot of options in it. If the problem persists, I would create a similar setup with empty config and try to configure only relevant options there (Minimal Working Example), to see if it works and it does not interfere with other options, access lists (just as example, it is extremely unlikely in this particular case) etc. If you do not have spare hardware, and your router is in production, so that you cannot experiment with empty configuration on it directly, you could:

Use Linux PCs/VMs with routing software like Quagga (mentioned in the question)
Use simulator from Cisco: Boson NetSim for CCNP supports BGP, however, I'm not sure if address-family/VPN/VRF are supported
Use virtual machines with IOS XRv from Cisco. As far as I remember, it was available for free with 2 Mbit/s bandwidth limit, which should be enough for testing. Again, I'm not sure if address-family/VPN/VRF are supported: Cisco IOS XRv router overview, VM download link
Use GNS3 (http://www.gns3.com/) simulator. There are Cisco IOS images for it, however, I do not know how to get them.
Finally, you could even try to buy used hardware from places like ebay as cheap as possible for testing purposes only.

Still can't confirm or disprove this theory. IOS version 15.1(2)SY4a also don't have options to place `dmzlink-bw` under `address-family`. I need some time to unravel cisco docs contradictions with real life. — Shamanu4, Nov 19 '15 at 13:01
It is really strange. You could try to look at following docs: http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/iproute_bgp/configuration/15-sy/irg-15-sy-book/irg-link-band.html, http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/iproute_bgp/configuration/15-sy/irg-15-sy-book/irg-multi-load.html, but I do not see anything missing in your config... — Andrey Sapegin, Nov 20 '15 at 10:33
I'm going to purchase cisco VIRL license and try to simulate topology there. I'll report about any results in post later. Thanks for help. — Shamanu4, Nov 22 '15 at 08:07

Cisco BGP Unequal Cost Load Balancing

1 Answers1