Layer 2 bridged networking issue with QEmu

Question

I am having a problem with a layer 2 bridged network configuration using Linux, QEmu, and VMware ESX. The configuration works fine on a small isolated network, but once introduced into our large corporate network, problems arise in the network layer.

I am using the server edition of Ubuntu 12.04.2 and QEmu 1.5.2 built from source emulating a SPARC host with qemu-system-sparc. The configuration I am trying to achieve is shown below with IPs shown where IPs are assigned and the last byte of the MAC addresses shown for the different interfaces.

    le0 (on QEmu hosted machine)
    192.168.1.103
    :56
    |
    tap0 (on Ubuntu machine)
    :7e                         
    |
    br0 (on Ubuntu machine)
    :19
    |
    eth1        eth0    (eth1 and eth0 on Ubuntu machine)
                192.168.1.100
    :19         :0f
    |           |
===========================
    Corporate Network
===========================
        |
        eth0
        192.168.1.102
        :84

The eth0 and eth1 interfaces are both created through VMware, which bridges them through a single physical NIC.

The /etc/network/interfaces file for the Ubuntu machine is shown below.

# The loopback network interface
auto lo
iface lo inet loopback

auto eth1
iface eth1 inet manual

auto tap0
iface tap0 inet manual
    pre-up tunctl -u my-user -t tap0
    up ip link set tap0 up

auto br0
iface br0 inet manual
    bridge_ports tap0 eth1
    bridge_fd 15
    bridge_hello 2
    bridge_maxage 20
    bridge_stp off
    bridge_waitport 60
    bridge_pathcost eth1 32768
    bridge_maxwait 0

# The primary network interface
auto eth0
iface eth0 inet static
    address 192.168.1.100
    netmask 255.255.255.0
    gateway 192.168.1.1

In both the isolated network and the corporate network ARP packets make it through the chain of bridges, and 192.168.1.102 has correct ARP information for 192.168.1.103 and vice versa. On the isolated network, the hosts can ping each other, and all seems well with application layer connections across the bridge.

However, on the corporate network, when pinging from the host 192.168.1.103 to 192.168.1.102 the ping packet makes it outward through the bridges to 192.168.1.102. The 192.168.1.102 host then attempts to send the ICMP reply, which never makes it back to the 192.168.1.103 host.

The ARP cache on 192.168.1.102 seems correct, with the IP address of 192.168.1.103 mapping to the :56 MAC address. However, if I run tcpdump on the eth1 interface of the Ubuntu machine I see the ICMP requests coming out from the 192.168.1.103 host, but don't see the replies coming back in (despite the fact that they are leaving 192.168.1.102 when dumped from eth0 on that machine).

I have STP off, though it is running on the corporate network. The output of brctl showstp br0 shows the bridge to be forwarding even when on the corporate network.

Any idea why this isn't working on a larger and more sophisticated network, but does work in isolation? Could our switching gear on the corporate network somehow have corrupt MAC address tables?

It's turned off on the bridge, but it's running on the larger corporate network. Even with it on, the IP traffic still only works in a single direction. — kgraney, Aug 06 '13 at 16:02

score 1 · Accepted Answer · answered Aug 08 '13 at 19:11

The solution to this problem was to enable promiscuous mode on the VMware ESX virtual switch as described here. The physical switch and Ubuntu network interfaces were both setup correctly, but VMware was preventing the incoming traffic from ever making it to the eth1 interface.

Layer 2 bridged networking issue with QEmu

1 Answers1