17

I have a linux box with 9 NICs on it, and I want eight of them to have unique addresses on the same subnet, e.g.:

ifconfig eth1 192.168.123.1 netmask 255.255.0.0
ifconfig eth2 192.168.123.2 netmask 255.255.0.0
ifconfig eth3 192.168.123.3 netmask 255.255.0.0
...
ifconfig eth8 192.168.123.8 netmask 255.255.0.0

The default ARP behaviour is extremely counterproductive in this case, since it results in all traffic for all IPs passing exclusively through eth1, which is pretty much the exact opposite of what I want.

So I rummaged around and ended up making some changes to sysctl such as this:

net.ipv4.conf.all.arp_filter=1
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2

That did prevent eth1 from impersonating all the others, but I still can't ping anything other than eth1's address successfully. (e.g. from a second computer on the same switch, only 192.168.123.1 responds to ping)

I'm guessing that I need to do something with arptables or iproute or SOMETHING, but I'm lost at sea in this field.

Bonus Points: Solution must be compatible with Linux 2.6.27.27. (More specifically, Slax 6.1.2)

frustrated_tester
  • 173
  • 1
  • 1
  • 4
  • Can you post the routing table? – ponsfonze Aug 08 '12 at 00:40
  • 2
    What is your purpose in creating this configuration? What are you trying to accomplish? – David Schwartz Aug 08 '12 at 01:03
  • this way madness lies. – Sirex Aug 08 '12 at 03:51
  • if you want to bond these interfaces you're going to have to actually bond them. – resmon6 Aug 08 '12 at 13:20
  • 1
    @DavidSchwartz I am testing some networking middleware which requires a homebrew test client. Basically the middleware and test client(tx) are on Box A, and test client(rx) is running on Boxes B, C, D, ...etc. Instead of having nine boxes with one NIC each (Boxes B, C, D, etc.), I want to have one box with nine NICs because it makes a lot of things ($$$, log collection, automation, etc) easier. Additionally, there are some bandwidth and other considerations which make virtual interfaces a poor choice. E.G. in my actual test setup I will have more than one "Box A". – frustrated_tester Aug 08 '12 at 20:12
  • So the 9 NICs are each going to connect to a different device and this machine is supposed to act like a switch? If so, why does each interface need a different IP address? – David Schwartz Aug 08 '12 at 21:04
  • 2
    @DavidSchwartz No, the machine is not supposed to act like a switch (or router). Think of it more as running 8 virtual machines with a dedicated physical NIC for each VM (_Note: I am_ not _running VMs, this is just an analogy_). From the point of view of another box on the same network, my single PC should be utterly indistinguishable from eight discrete PCs. – frustrated_tester Aug 08 '12 at 22:17
  • If it is to be utterly indistinguishable from eight discrete PCs, then VMs is you actual answer. Maybe, and only maybe, LXCs could accomplish this, but I have significant doubts since they are still sharing the same one IP stack. If you run a process on this one machine, which of the utterly indistinguishable eight discrete PCs is it to be effectively running on? – Skaperen Aug 09 '12 at 02:54

5 Answers5

19

You need a strong end system model. Linux is fundamentally built around a weak send system model, so it's really not a good OS choice for this application.

You will have to fake every piece of the behavior you need, from ARP to policy routing to source address selection. You will also need filters to prevent packets from being accepted if they arrive on the wrong interface.

The definitely necessary steps are:

  1. Configure arp_filter=1 and arp_ignore=2 on all interfaces.

  2. Add per-interface, source-based routing for outgoing traffic. (Destination interface must be chosen based on source address.)

  3. Add per-interface ingress filtering to silently drop packets received on the wrong interface. (Packets with a destination address assigned to another interface.)

Unfortunately, there is no consensus on whether these three steps are all that is needed. The weak end system model is built into the entire Linux TCP/IP stack, and it's not clear what might go wrong with subtle issues like multicast.

It's not clear how you would choose the output interface for broadcasts, for example. Should it go out all of them? Maybe. What is the right behavior if the stack gets an outbound broadcast with a source address not assigned to one of the interfaces?

Again, you have chosen the wrong tool for the job.

David Schwartz
  • 31,215
  • 2
  • 53
  • 82
7

You more likely want to create a bridge with the 8/9 interfaces and then assign an IP address to that bridge (bridge-utils packet, command 'brctl add').

This way the bridge will act like as a switch and can have an IP address into your subnet.

Be2eB
  • 71
  • 1
  • 1
  • AFAIK, this should actually be the correct answer for Linux. Linux gets around many of the problems mentioned in David's answer by using bridge devices (aka avoid L3 problems by building up a better L2 logical network). – Dave Apr 26 '16 at 01:15
4

I would recommend bonding the physical interfaces then configure all of the addresses on the single bonded interface.

You'll need support on the switch as well.

Here's a mini tutorial that you can use to get started.

bahamat
  • 6,193
  • 23
  • 28
4

It seems you want a test environment equivalent to 9 separate machines, and believed that 9 interfaces on one machine could emulate that. In Linux it simply cannot do this through a single stack for reasons David Schwartz described. BTDT and have the scars. It was bad enough with 2 interfaces.

A better solution might be to run 8 or 9 discrete virtual machines in your one host, and bridge 8 or 9 interfaces to these virtual machines.

Skaperen
  • 1,064
  • 2
  • 11
  • 21
  • Reading resources: Unix Network Programming: The Sockets Networking API, by Stevens, Fenner, and Rudoff. Also see RFC1122 and RFC4907. – Skaperen Aug 09 '12 at 01:40
0

Yes it is possible following David Schwartz suggestion:

echo -ne 0 > /proc/sys/net/ipv4/conf/all/rp_filter
echo -ne 0 > /proc/sys/net/ipv4/conf/eth0/rp_filter
echo -ne 0 > /proc/sys/net/ipv4/conf/eth3/rp_filter

//For proper functionality i.e. ARP replies from eth1 to get generated when both eth0 and eth1 are in same subnet

echo -ne 0 > /proc/sys/net/ipv4/conf/all/arp_filter
echo -ne 2 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo -ne 0 > /proc/sys/net/ipv4/conf/eth0/arp_filter
echo -ne 2 > /proc/sys/net/ipv4/conf/eth0/arp_ignore
echo -ne 0 > /proc/sys/net/ipv4/conf/eth1/arp_filter
echo -ne 2 > /proc/sys/net/ipv4/conf/eth1/arp_ignore

//Create a table called "new_rt_table" and create a routing rule that says any packet with a mark equal to '1' gets routed according to the "new_rt_table"(can name it whatever you want) table. The file /etc/iproute2/rt_tables is the only source of table names on the system. Internally, routing tables have integer identifiers.

echo 1 new_rt_table >> /etc/iproute2/rt_tables
ip rule add from all fwmark 1 table new_rt_table

//setup the "new_rt_table" table to route the packets via eth1

ip route add default dev eth1 table new_rt_table
ip route show table new_rt_table

//mark packets so that 'ip route' can route it through eth1

iptables -F -t mangle
iptables -t mangle -I OUTPUT -s <ip addr of eth1> -o eth0 -j MARK --set-mark 1

//enable support for multiple routing tables in kernel config.

Kernel Configuration

→ Networking support → Networking options

[*] IP: advanced router

[*] IP: policy routing

CONFIG_IP_ADVANCED_ROUTER

CONFIG_IP_MULTIPLE_TABLES

//the above steps redirects packets meant to be output from eth0 to properly exit from eth1.

Kindly suggest any other methods if anyone gets it working.

noel av
  • 1
  • 1