1

Short story: I've got a faulty cisco 1811 on my hands. It's a dual wan fixed configuration router. I had it setup with route maps to do load balancing. It works but every couple of hours it shits itself and needs rebooting. I need to take it out of it's current position on the network.

So what I've done (for temporary purposes) is built up a new box, installed debian, iproute2 and configured iptables.

I've got two internet connections, both on separate phone lines, both to separate linksys routers. The routers are in bridged mode. The new box has 3 NICS (2 nics for connecting to each router in bridge mode) and one for internal network. I've used pppoeconf to setup two new pppoe connections. That all works.

It seems to work when only one connection is up, but not when both are up. I have a feeling that it doesn't know which interface to process the packet on. I've read many documents about split access and load balancing but I can't get it working.

gateway:~# ifconfig -a
eth1      Link encap:Ethernet  HWaddr 00:24:8c:4e:31:e7
         inet addr:192.168.5.254  Bcast:192.168.5.255  Mask:255.255.255.0
         inet6 addr: fe80::224:8cff:fe4e:31e7/64 Scope:Link
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:3494 errors:0 dropped:0 overruns:0 frame:0
         TX packets:3553 errors:0 dropped:0 overruns:0 carrier:1
         collisions:0 txqueuelen:1000
         RX bytes:633209 (618.3 KiB)  TX bytes:1668656 (1.5 MiB)
         Interrupt:219

eth2      Link encap:Ethernet  HWaddr 00:40:f4:ba:c4:41
         inet addr:192.168.1.2  Bcast:192.168.1.255  Mask:255.255.255.0
         inet6 addr: fe80::240:f4ff:feba:c441/64 Scope:Link
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:365 errors:0 dropped:0 overruns:0 frame:0
         TX packets:377 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:57190 (55.8 KiB)  TX bytes:85824 (83.8 KiB)
         Interrupt:16 Base address:0xe800

eth3      Link encap:Ethernet  HWaddr 00:60:67:72:40:42
         inet addr:192.168.1.1  Bcast:192.168.1.255  Mask:255.255.255.0
         inet6 addr: fe80::260:67ff:fe72:4042/64 Scope:Link
         UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
         RX packets:2287 errors:0 dropped:0 overruns:0 frame:0
         TX packets:1971 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:1000
         RX bytes:1414715 (1.3 MiB)  TX bytes:425185 (415.2 KiB)
         Interrupt:17 Base address:0xe400

lo        Link encap:Local Loopback
         inet addr:127.0.0.1  Mask:255.0.0.0
         inet6 addr: ::1/128 Scope:Host
         UP LOOPBACK RUNNING  MTU:16436  Metric:1
         RX packets:150 errors:0 dropped:0 overruns:0 frame:0
         TX packets:150 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:0
         RX bytes:15596 (15.2 KiB)  TX bytes:15596 (15.2 KiB)

ppp0      Link encap:Point-to-Point Protocol
         inet addr:150.101.177.X  P-t-P:150.101.180.24  Mask:255.255.255.255
         UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1492  Metric:1
         RX packets:1955 errors:0 dropped:0 overruns:0 frame:0
         TX packets:1446 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:3
         RX bytes:1344970 (1.2 MiB)  TX bytes:353725 (345.4 KiB)

ppp1      Link encap:Point-to-Point Protocol
         inet addr:150.101.179.X  P-t-P:150.101.180.143  Mask:255.255.255.255
         UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1492  Metric:1
         RX packets:28 errors:0 dropped:0 overruns:0 frame:0
         TX packets:37 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:3
         RX bytes:6691 (6.5 KiB)  TX bytes:10285 (10.0 KiB)

I can dial both connections and they both answer and authenticate successfully.

gateway:~# cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
allow-hotplug eth1
iface eth1 inet static
address 192.168.5.254
netmask 255.255.255.0
broadcast 192.168.5.255

auto eth3
allow-hotplug eth3
iface eth3 inet dhcp

auto eth2
allow-hotplug eth2
iface eth2 inet dhcp


auto dsl-provider1
iface dsl-provider1 inet ppp
pre-up /sbin/ifconfig eth3 up # line maintained by pppoeconf
#post-up ip route add 150.101.180.143 dev eth1 src 150.101.179.X table T2
#post-up ip route add default via 150.101.180.143 table T2
#post-up ip rule add from 150.101.179.X table T2
#post-down ip rule del from 150.101.179.X table T2
provider dsl-provider1

auto dsl-provider
iface dsl-provider inet ppp
pre-up /sbin/ifconfig eth2 up # line maintained by pppoeconf
#post-up ip route add 150.101.180.24 dev eth2 src 150.101.177.X table T1
#post-up ip route add default via 150.101.180.24 table T1
#post-up ip rule add from 150.101.177.X table T1
#post-down ip rule del from 150.101.177.X table T1
provider dsl-provider


gateway:~#

As you can see above I've commented out the post-up's and down's to set the ip routes Should I be using ethX as the dev in ip route commands? or should it be pppX? I also note that when any interface changes, the ip routes are refreshed and changes lost. And here is my rt_tables file.

gateway:~# cat /etc/iproute2/rt_tables
#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
200     uplink1
201     uplink2

100     T1
101     T2
gateway:~#

This is my ip tables script. I know it needs changing, (especially the default rules) but that aside: - It works after one connection is up, and

#!/bin/sh
INTIF="eth1"

EXTIF="ppp0"
EXTIF2="ppp1"

EXTIP="`/sbin/ifconfig ppp0 | grep 'inet addr' | awk '{print $2}' | sed -e 's/.*://'`"
EXTIP2="`/sbin/ifconfig ppp1 | grep 'inet addr' | awk '{print $2}' | sed -e 's/.*://'`"

/sbin/depmod -a
/sbin/modprobe ip_tables
/sbin/modprobe ip_conntrack
/sbin/modprobe ip_conntrack_ftp
/sbin/modprobe ip_conntrack_irc
/sbin/modprobe iptable_nat
/sbin/modprobe ip_nat_ftp
echo "1" > /proc/sys/net/ipv4/ip_forward
echo "1" > /proc/sys/net/ipv4/ip_dynaddr
iptables -P INPUT ACCEPT
iptables -F INPUT
iptables -P OUTPUT ACCEPT
iptables -F OUTPUT
iptables -P FORWARD DROP
iptables -F FORWARD
iptables -t nat -F

iptables -A FORWARD -i $EXTIF -o $INTIF -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT
iptables -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE

iptables -A FORWARD -i $EXTIF2 -o $INTIF -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -i $INTIF -o $EXTIF2 -j ACCEPT
iptables -t nat -A POSTROUTING -o $EXTIF2 -j MASQUERADE


#Transparent Proxy
iptables -t nat -A PREROUTING -i $INTIF -p tcp --dport 80 -j REDIRECT --to-port 3128

#Mail
iptables -A FORWARD -i $EXTIF -o $INTIF -d 192.168.5.4 -p tcp --dport 25 -j ACCEPT
iptables -t nat -A PREROUTING -i $EXTIF -d $EXTIP -p tcp --dport 25 -j DNAT --to 192.168.5.4
iptables -A FORWARD -i $EXTIF2 -o $INTIF -d 192.168.5.4 -p tcp --dport 25 -j ACCEPT
iptables -t nat -A PREROUTING -i $EXTIF2 -d $EXTIP2 -p tcp --dport 25 -j DNAT --to 192.168.5.4

#HTTPS (webmail)
iptables -A FORWARD -i $EXTIF -o $INTIF -d 192.168.5.4 -p tcp --dport 443 -j ACCEPT
iptables -t nat -A PREROUTING -i $EXTIF -d $EXTIP -p tcp --dport 443 -j DNAT --to 192.168.5.4
iptables -A FORWARD -i $EXTIF2 -o $INTIF -d 192.168.5.4 -p tcp --dport 443 -j ACCEPT
iptables -t nat -A PREROUTING -i $EXTIF2 -d $EXTIP2 -p tcp --dport 443 -j DNAT --to 192.168.5.4

#IMAPS
iptables -A FORWARD -i $EXTIF -o $INTIF -d 192.168.5.4 -p tcp --dport 993 -j ACCEPT
iptables -t nat -A PREROUTING -i $EXTIF -d $EXTIP -p tcp --dport 993 -j DNAT --to 192.168.5.4
iptables -A FORWARD -i $EXTIF2 -o $INTIF -d 192.168.5.4 -p tcp --dport 993 -j ACCEPT
iptables -t nat -A PREROUTING -i $EXTIF2 -d $EXTIP2 -p tcp --dport 993 -j DNAT --to 192.168.5.4

#POP3
iptables -A FORWARD -i $EXTIF -o $INTIF -d 192.168.5.4 -p tcp --dport 110 -j ACCEPT
iptables -t nat -A PREROUTING -i $EXTIF -d $EXTIP -p tcp --dport 110 -j DNAT --to 192.168.5.4
iptables -A FORWARD -i $EXTIF2 -o $INTIF -d 192.168.5.4 -p tcp --dport 110 -j ACCEPT
iptables -t nat -A PREROUTING -i $EXTIF2 -d $EXTIP2 -p tcp --dport 110 -j DNAT --to 192.168.5.4

#VPN
iptables -A FORWARD -i $EXTIF -o $INTIF -d 192.168.5.4 -p tcp --dport 1723 -j ACCEPT
iptables -t nat -A PREROUTING -i $EXTIF -d $EXTIP -p tcp --dport 1723 -j DNAT --to 192.168.5.4
iptables -A FORWARD -i $EXTIF2 -o $INTIF -d 192.168.5.4 -p tcp --dport 1723 -j ACCEPT
iptables -t nat -A PREROUTING -i $EXTIF2 -d $EXTIP2 -p tcp --dport 1723 -j DNAT --to 192.168.5.4

#HTTP
iptables -A FORWARD -i $EXTIF -o $INTIF -d 192.168.5.4 -p tcp --dport 80 -j ACCEPT
iptables -t nat -A PREROUTING -i $EXTIF -d $EXTIP -p tcp --dport 80 -j DNAT --to 192.168.5.4
iptables -A FORWARD -i $EXTIF2 -o $INTIF -d 192.168.5.4 -p tcp --dport 80 -j ACCEPT
iptables -t nat -A PREROUTING -i $EXTIF2 -d $EXTIP2 -p tcp --dport 80 -j DNAT --to 192.168.5.4

#Bridgit
iptables -A FORWARD -i $EXTIF -o $INTIF -d 192.168.5.4 -p tcp --dport 8080 -j ACCEPT
iptables -t nat -A PREROUTING -i $EXTIF -d $EXTIP -p tcp --dport 8080 -j DNAT --to 192.168.5.4
iptables -A FORWARD -i $EXTIF2 -o $INTIF -d 192.168.5.4 -p tcp --dport 8080 -j ACCEPT
iptables -t nat -A PREROUTING -i $EXTIF2 -d $EXTIP2 -p tcp --dport 8080 -j DNAT --to 192.168.5.4

#GATEWAY2="150.101.180.143"
#GATEWAY="150.101.180.24"

#ip route add $GATEWAY dev $EXTIF src $EXTIP table T1
#ip route add default via $GATEWAY table T1
#ip route add $GATEWAY2 dev $EXTIF2 src $EXTIP2 table T2
#ip route add default via $GATEWAY2 table T2

#ip rule add from $EXTIP table T1
#ip rule add from $EXTIP2 table T2

#ip route add default scope global nexthop via $GATEWAY dev $EXTIF weight 1 nexthop via $GATEWAY2 dev $EXTIF2 weight 1

Any ideas would be fantastic.

Thanks,

Mike

EDIT: Here is the requested commands and output which were requested from the comments on this question. Thanks!

Explanation: I've turned on the machine with both interfaces automatically coming up on bootup (ppp dials and connects automatically). You can see as soon as I turn off one interface it works.

login as: root
root@192.168.5.254's password:
Linux gateway 2.6.26-2-686 #1 SMP Thu Sep 16 19:35:51 UTC 2010 i686

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Wed Nov  3 04:07:28 2010 from 192.168.5.38
gateway:~# ping google.com
ping: unknown host google.com
gateway:~# ip route
150.101.180.24 dev ppp0  proto kernel  scope link  src 150.101.179.XXX
150.101.180.24 dev ppp1  proto kernel  scope link  src 150.101.177.XXX
192.168.5.0/24 dev eth1  proto kernel  scope link  src 192.168.5.254
192.168.1.0/24 dev eth2  proto kernel  scope link  src 192.168.1.2
192.168.1.0/24 dev eth3  proto kernel  scope link  src 192.168.1.3
default via 192.168.1.254 dev eth3
default via 192.168.1.254 dev eth2
default dev ppp1  scope link
default dev ppp0  scope link
gateway:~# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 10                                                                                     00
    link/ether 00:24:8c:4e:31:e7 brd ff:ff:ff:ff:ff:ff
3: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN ql                                                                                     en 1000
    link/ether 00:40:f4:ba:c4:41 brd ff:ff:ff:ff:ff:ff
4: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN ql                                                                                     en 1000
    link/ether 00:60:67:72:40:42 brd ff:ff:ff:ff:ff:ff
5: ppp0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qdisc pfifo_fast state UN                                                                                     KNOWN qlen 3
    link/ppp
6: ppp1: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qdisc pfifo_fast state UN                                                                                     KNOWN qlen 3
    link/ppp
gateway:~# clear
gateway:~# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:24:8c:4e:31:e7 brd ff:ff:ff:ff:ff:ff
3: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 00:40:f4:ba:c4:41 brd ff:ff:ff:ff:ff:ff
4: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 00:60:67:72:40:42 brd ff:ff:ff:ff:ff:ff
5: ppp0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qdisc pfifo_fast state UNKNOWN qlen 3
    link/ppp
6: ppp1: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qdisc pfifo_fast state UNKNOWN qlen 3
    link/ppp
gateway:~# ip neigh show
192.168.5.38 dev eth1 lladdr 00:1d:60:0a:cb:23 DELAY
192.168.1.254 dev eth3 lladdr 00:04:ed:a9:07:81 REACHABLE
192.168.1.254 dev eth2 lladdr 00:04:ed:aa:cd:45 REACHABLE
192.168.5.43 dev eth1 lladdr 00:08:5d:03:b3:3f STALE
192.168.5.4 dev eth1 lladdr 00:1e:0b:1c:4e:60 STALE
gateway:~# ip route get 5.5.5.5
5.5.5.5 via 192.168.1.254 dev eth3  src 192.168.1.3
    cache  mtu 1500 advmss 1460 hoplimit 64
gateway:~# ping google.com
ping: unknown host google.com
gateway:~# ping 4.2.2.2
PING 4.2.2.2 (4.2.2.2) 56(84) bytes of data.
From 192.168.1.254 icmp_seq=1 Destination Net Unreachable
From 192.168.1.254 icmp_seq=2 Destination Net Unreachable
^C
--- 4.2.2.2 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1003ms

gateway:~# ifconfig -a
eth1      Link encap:Ethernet  HWaddr 00:24:8c:4e:31:e7
          inet addr:192.168.5.254  Bcast:192.168.5.255  Mask:255.255.255.0
          inet6 addr: fe80::224:8cff:fe4e:31e7/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:739 errors:0 dropped:0 overruns:0 frame:0
          TX packets:303 errors:0 dropped:0 overruns:0 carrier:2
          collisions:0 txqueuelen:1000
          RX bytes:59730 (58.3 KiB)  TX bytes:35821 (34.9 KiB)
          Interrupt:219

eth2      Link encap:Ethernet  HWaddr 00:40:f4:ba:c4:41
          inet addr:192.168.1.2  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::240:f4ff:feba:c441/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:152 errors:0 dropped:0 overruns:0 frame:0
          TX packets:119 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:15328 (14.9 KiB)  TX bytes:8207 (8.0 KiB)
          Interrupt:16 Base address:0xe800

eth3      Link encap:Ethernet  HWaddr 00:60:67:72:40:42
          inet addr:192.168.1.3  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::260:67ff:fe72:4042/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:521 errors:0 dropped:0 overruns:0 frame:0
          TX packets:758 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:48899 (47.7 KiB)  TX bytes:57096 (55.7 KiB)
          Interrupt:17 Base address:0xe400

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:20 errors:0 dropped:0 overruns:0 frame:0
          TX packets:20 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1160 (1.1 KiB)  TX bytes:1160 (1.1 KiB)

ppp0      Link encap:Point-to-Point Protocol
          inet addr:150.101.179.XXX  P-t-P:150.101.180.24  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1492  Metric:1
          RX packets:61 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:3
          RX bytes:7454 (7.2 KiB)  TX bytes:54 (54.0 B)

ppp1      Link encap:Point-to-Point Protocol
          inet addr:150.101.177.XXX  P-t-P:150.101.180.24  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1492  Metric:1
          RX packets:127 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:3
          RX bytes:6278 (6.1 KiB)  TX bytes:54 (54.0 B)

gateway:~# ifdown eth2
There is already a pid file /var/run/dhclient.eth2.pid with pid 2217
killed old client process, removed PID file
Internet Systems Consortium DHCP Client V3.1.1
Copyright 2004-2008 Internet Systems Consortium.
All rights reserved.
For info, please visit http://www.isc.org/sw/dhcp/

Listening on LPF/eth2/00:40:f4:ba:c4:41
Sending on   LPF/eth2/00:40:f4:ba:c4:41
Sending on   Socket/fallback
DHCPRELEASE on eth2 to 192.168.1.254 port 67
gateway:~# ping google.com
PING google.com (66.102.11.104) 56(84) bytes of data.
64 bytes from syd01s01-in-f104.1e100.net (66.102.11.104): icmp_seq=1 ttl=58 time=32.8 ms
64 bytes from syd01s01-in-f104.1e100.net (66.102.11.104): icmp_seq=2 ttl=58 time=32.7 ms
64 bytes from syd01s01-in-f104.1e100.net (66.102.11.104): icmp_seq=3 ttl=58 time=32.9 ms
64 bytes from syd01s01-in-f104.1e100.net (66.102.11.104): icmp_seq=4 ttl=57 time=33.3 ms
64 bytes from syd01s01-in-f104.1e100.net (66.102.11.104): icmp_seq=5 ttl=57 time=32.6 ms
^C
--- google.com ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4017ms
rtt min/avg/max/mdev = 32.694/32.923/33.373/0.263 ms
gateway:~# ifup eth2
Internet Systems Consortium DHCP Client V3.1.1
Copyright 2004-2008 Internet Systems Consortium.
All rights reserved.
For info, please visit http://www.isc.org/sw/dhcp/

Listening on LPF/eth2/00:40:f4:ba:c4:41
Sending on   LPF/eth2/00:40:f4:ba:c4:41
Sending on   Socket/fallback
DHCPDISCOVER on eth2 to 255.255.255.255 port 67 interval 8
DHCPOFFER from 192.168.1.254
DHCPREQUEST on eth2 to 255.255.255.255 port 67
DHCPACK from 192.168.1.254
bound to 192.168.1.2 -- renewal in 20576 seconds.
if-up.d/mountnfs[eth2]: waiting for interface dsl-provider1 before doing NFS mounts (warning).
if-up.d/mountnfs[eth2]: waiting for interface dsl-provider before doing NFS mounts (warning).
gateway:~# ping google.com
PING google.com (66.102.11.104) 56(84) bytes of data.
^C
--- google.com ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4013ms

gateway:~# ifdown eth3
There is already a pid file /var/run/dhclient.eth3.pid with pid 2523
killed old client process, removed PID file
Internet Systems Consortium DHCP Client V3.1.1
Copyright 2004-2008 Internet Systems Consortium.
All rights reserved.
For info, please visit http://www.isc.org/sw/dhcp/

Listening on LPF/eth3/00:60:67:72:40:42
Sending on   LPF/eth3/00:60:67:72:40:42
Sending on   Socket/fallback
DHCPRELEASE on eth3 to 192.168.1.254 port 67
gateway:~# ping google.com
ping: unknown host google.com
gateway:~# ping 4.2.2.2
connect: Network is unreachable
gateway:~# ping 4.2.2.2
connect: Network is unreachable
gateway:~# ip route
150.101.180.24 dev ppp0  proto kernel  scope link  src 150.101.179.XXX
192.168.5.0/24 dev eth1  proto kernel  scope link  src 192.168.5.254
192.168.1.0/24 dev eth2  proto kernel  scope link  src 192.168.1.2
gateway:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
150.101.180.24  *               255.255.255.255 UH    0      0        0 ppp0
192.168.5.0     *               255.255.255.0   U     0      0        0 eth1
192.168.1.0     *               255.255.255.0   U     0      0        0 eth2
gateway:~# ping google.com
ping: unknown host google.com
gateway:~# plog
Nov 11 17:17:07 gateway pppd[2139]: sendPacket: send: Network is down
Nov 11 17:17:07 gateway pppd[2139]: Exit.
gateway:~# poff -a
gateway:~# pon dsl-provider
Plugin rp-pppoe.so loaded.
gateway:~# pon dsl-provider1
Plugin rp-pppoe.so loaded.
gateway:~# ping google.com
PING google.com (66.102.11.104) 56(84) bytes of data.
64 bytes from syd01s01-in-f104.1e100.net (66.102.11.104): icmp_seq=1 ttl=57 time=32.8 ms
64 bytes from syd01s01-in-f104.1e100.net (66.102.11.104): icmp_seq=2 ttl=58 time=32.5 ms
64 bytes from syd01s01-in-f104.1e100.net (66.102.11.104): icmp_seq=3 ttl=57 time=32.9 ms
64 bytes from syd01s01-in-f104.1e100.net (66.102.11.104): icmp_seq=4 ttl=57 time=32.3 ms
^C
--- google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3008ms
rtt min/avg/max/mdev = 32.373/32.657/32.906/0.218 ms
gateway:~#
gateway:~# ip route get 5.5.5.5
5.5.5.5 dev ppp0  src 150.101.179.XXX
    cache  mtu 1492 advmss 1452 hoplimit 64
gateway:~#
Mike
  • 196
  • 1
  • 2
  • 13
  • 1
    What errors do you get when it's not working? Can you post `ip link show; ip neigh show; ip route show; ip route get 5.5.5.5;` from when both links are up? – al. Nov 04 '10 at 22:25
  • I will run these commands this afternoon after it is not in use, and will post the results here. – Mike Nov 05 '10 at 04:03
  • Sorry for the delay. I have attached an edit to the question with those commands. Thanks for your help – Mike Nov 10 '10 at 21:34

1 Answers1

1

As you can see here

  gateway:~# ip route
  150.101.180.24 dev ppp0  proto kernel  scope link  src 150.101.179.XXX
  150.101.180.24 dev ppp1  proto kernel  scope link  src 150.101.177.XXX
  192.168.5.0/24 dev eth1  proto kernel  scope link  src 192.168.5.254
  192.168.1.0/24 dev eth2  proto kernel  scope link  src 192.168.1.2
  192.168.1.0/24 dev eth3  proto kernel  scope link  src 192.168.1.3
> default via 192.168.1.254 dev eth3
> default via 192.168.1.254 dev eth2
> default dev ppp1  scope link
> default dev ppp0  scope link

you have 4 default routes. When you're trying to reach a non-local host, one of these 4 is chosen:

gateway:~# ip route get 5.5.5.5
5.5.5.5 via 192.168.1.254 dev eth3  src 192.168.1.3
    cache  mtu 1500 advmss 1460 hoplimit 64

In this case eth3, which doesn't get you to the desired destination. The gateway you're asking to forward your packets (192.168.1.254) truthfully responds, that it has no path to the destination network:

gateway:~# ping 4.2.2.2
PING 4.2.2.2 (4.2.2.2) 56(84) bytes of data.
From 192.168.1.254 icmp_seq=1 Destination Net Unreachable

I don't know where those default routes via ethX originate from, but you need to get rid of them:

ip route del default via 192.168.1.254 dev eth3
ip route del default via 192.168.1.254 dev eth2

If your setup is not working as intended after that, the output of ip rule show will probably be helpful for further debugging.

al.
  • 915
  • 6
  • 17
  • Why is it when I pull only one interface down, that it starts working? I don't touch any routes, besides pulling down the interface, which, would mean that one of the default routes via 192.168.1.254 would be there. – Mike Nov 10 '10 at 22:58
  • I can try removing those routes this afternoon, and running ip rule show and let you know the output. thanks for your help so far, al! – Mike Nov 10 '10 at 22:59
  • My guess would be dhclient, but I didn't really pay attention to that, as it is a separate problem. – al. Nov 10 '10 at 23:10