qemu-kvm/virsh: No network connectivity whilst using bridged networking

Question

I'm having an issue with performing virtualisation via libvirt/qemu-kvm, in which my set up for bridged networking doesn't appear to be working correctly. I've followed every tutorial I can find and spent hours and hours going through forums, but currently to no avail.

Both the host and guest are using Centos6.

Here's the set up for the host:

[root@node2 ~]# ifconfig -a
bridge0   Link encap:Ethernet  HWaddr BC:30:5B:E8:C0:4F  
          inet addr:10.59.190.253  Bcast:10.59.190.255  Mask:255.255.255.0
          inet6 addr: fe80::be30:5bff:fee8:c04f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5315 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3350 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:316961 (309.5 KiB)  TX bytes:3943630 (3.7 MiB)

eth0      Link encap:Ethernet  HWaddr BC:30:5B:E8:C0:4F  
          inet6 addr: fe80::be30:5bff:fee8:c04f/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:5286 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5325 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:409939 (400.3 KiB)  TX bytes:4095006 (3.9 MiB)
          Interrupt:16 Memory:c0000000-c0012800 

eth1      Link encap:Ethernet  HWaddr BC:30:5B:E8:C0:50  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
          Interrupt:17 Memory:c2000000-c2012800 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:28 errors:0 dropped:0 overruns:0 frame:0
          TX packets:28 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:2832 (2.7 KiB)  TX bytes:2832 (2.7 KiB)

sit0      Link encap:IPv6-in-IPv4  
          NOARP  MTU:1480  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

vnet0     Link encap:Ethernet  HWaddr FE:54:00:3A:C5:C9  
          inet6 addr: fe80::fc54:ff:fe3a:c5c9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:492 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:0 (0.0 b)  TX bytes:29516 (28.8 KiB)

I have two physical adapters, eth0 and eth1. eth1 is currently unplugged and not being used in this set up. eth0 is bridged to bridge0. bridge0 is using DHCP to get its network configuration. The adapters use the following configurations:

[root@node2 network-scripts]# cat ifcfg-eth0
DEVICE="eth0"
NM_CONTROLLED="no"
ONBOOT=yes
HWADDR=BC:30:5B:E8:C0:4F
BRIDGE=bridge0
[root@node2 network-scripts]# cat ifcfg-bridge0
DEVICE=bridge0
ONBOOT=yes
TYPE=Bridge
BOOTPROTO=dhcp
DELAY=0
NM_CONTROLLED=no

...and iptables set up as follows:

[root@node2 network-scripts]# iptables --list-rules
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT 
-A INPUT -p icmp -j ACCEPT 
-A INPUT -i lo -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT 
-A INPUT -p tcp -m state --state NEW -m tcp --dport 5900 -j ACCEPT 
-A INPUT -j REJECT --reject-with icmp-host-prohibited 
-A FORWARD -m physdev --physdev-is-bridged -j ACCEPT 
-A FORWARD -m physdev --physdev-is-bridged -j ACCEPT 
-A FORWARD -j REJECT --reject-with icmp-host-prohibited

So, that's the set up. I then used the following command to create and install a Centos6 guest image:

virt-install --name=centos-bridge-test --disk path=/var/lib/libvirt/images/centos6-minimal,size=5 --vnc --ram=512 --cdrom=/var/lib/libvirt/images/CentOS-6.0-x86_64-minimal.iso --vnclisten=0.0.0.0 -b bridge0

My understanding is that this set-up should have been sufficient for the bridged networking to "just work," however, when I start the guest using the virsh command (virsh start centos-bridge-test) it has no network connectivity.

Here's an xml dump of the libvirt domain:

[root@node2 network-scripts]# virsh dumpxml centos-bridge-test
<domain type='kvm' id='7'>
  <name>centos-bridge-test</name>
  <uuid>5d611267-2feb-c6f7-7a48-29f9695a4a75</uuid>
  <memory>524288</memory>
  <currentMemory>524288</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch='x86_64' machine='rhel6.0.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/var/lib/libvirt/images/centos6-minimal'/>
      <target dev='hda' bus='ide'/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' unit='0'/>
    </disk>
    <disk type='block' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <alias name='ide0-1-0'/>
      <address type='drive' controller='0' bus='1' unit='0'/>
    </disk>
    <controller type='ide' index='0'>
      <alias name='ide0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:3a:c5:c9'/>
      <source bridge='bridge0'/>
      <target dev='vnet0'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/1'>
      <source path='/dev/pts/1'/>
      <target port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0' keymap='en-gb'/>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='selinux'>
    <label>system_u:system_r:svirt_t:s0:c384,c608</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c384,c608</imagelabel>
  </seclabel>
</domain>

The interface appears to be set up correctly (from looking at the libvirt documentation).

So, whenever I start the guest, there is no network connectivity.

[root@localhost ~]# ifconfig -a
eth0       Link encap:Ehternet  HWaddr 52:54:00:3A:C5:C9
           BROADCAST MULTICAST  MTU:1500  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000 
           RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
           Interrupt:10 Base address:0x6000

lo         Link encap:Local Loopback  
           inet addr:127.0.0.1  Mask:255.0.0.0
           inet6 addr: ::1/128 Scope:Host
           UP LOOPBACK RUNNING  MTU:16436  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0 
           RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

Edit: Note that eth0 is not up here, so I've been running ifup eth0 to get the adapter to come up, which then gives the following output:

[root@localhost ~]# ifconfig -a
eth0       Link encap:Ehternet  HWaddr 52:54:00:3A:C5:C9
           inet6 addr: fe80::5054:ff:fe3a:c5c9/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets65 errors:0 dropped:0 overruns:0 frame:0
           TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000 
           RX bytes:3896 (3.8 KiB)  TX bytes:398 (398.0 b)

lo         Link encap:Local Loopback  
           inet addr:127.0.0.1  Mask:255.0.0.0
           inet6 addr: ::1/128 Scope:Host
           UP LOOPBACK RUNNING  MTU:16436  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0 
           RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

...and when I attempt to access the network for either an internal or external address, I get this:

[root@localhost ~]# ping 10.59.190.253
connect: Network is unreachable
[root@localhost ~]# ping www.google.com
ping: unknown host www.google.com
[root@localhost ~]# ping 173.194.66.99
connect: Network is unreachable

Additional Info: The following is some additional information requested form the comments for this question. These commands were run from the host with the guest currently running:

[root@node2 network-scripts]# brctl show
bridge name         bridge id               STP enabled     interfaces
bridge0             8000.bc305be8c04f       no              eth0
                                                            vnet0
[root@node2 network-scripts]# ps -ef | grep qemu
qemu      1597     1  0 11:28 ?        00:00:25 /usr/libexec/qemu-kvm -S -M rhel6.0.0 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name centos-bridge-test -uuid 5d611267-2feb-c6f7-7a48-29f9695a4a75 -nodefconfig -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/centos-bridge-test.monitor,server,nowait -mon chardev=monitor,mode=control -rtc base=utc -boot c -drive file=/var/lib/libvirt/images/centos6-minimal,if=none,id=drive-ide0-0-0,boot=on,format=raw,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=20,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:3a:c5:c9,bus=pci.0,addr=0x3 -chardev pty,id=serial0 -device isa-serial,chardev=serial0 -usb -vnc 0.0.0.0:0 -k en-gb -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
root      2815 32057  0 14:31 pts/0    00:00:00 grep qemu

I'd really appreciate any help possible at the moment! If you need me to provide any further configurations, etc, please do let me know.

Edit: I'm using a minimal install of Centos6, so I don't have any window manager installed, so I'm trying to do everything via virsh and virsh-install, and would like to try to avoid using virsh-manager. Thanks.

good job posting all this data, but there's another thing required: start the VM up, and get `brctl show` and `ps -ef |grep qemu` — dyasny, Nov 21 '11 at 12:23
@dyasny Thanks for letting me know. I've added that information to the question. — a.pollock, Nov 21 '11 at 14:27
@dyasny I've edited the question to add the output from _ifconfig_ on the guest after running _ifup eth0_. This should provide some more information. — a.pollock, Nov 21 '11 at 15:21

score 3 · Accepted Answer · answered Nov 22 '11 at 10:47

3

The answer to this issue was that because I'd done a non-graphic install of Centos 6 minimal on the guests, the network interfaces had not been automatically set up to work. My lack of knowledge of virtualisation and Centos meant that I assumed that this would be taken care of for me, but I was incorrect in this assumption.

Fix: I needed to go into ifcfg-eth0 on the guests and turn on ONBOOT=yes and BOOTPROTO=dhcp, then restart the network adapter via ifdown eth0 then ifup eth0. After doing this, the guests received IP addresses and everything worked as expected.

answered Nov 22 '11 at 10:47

a.pollock

91
1
7

1

oh wow, it's like checking the power cable for a server that just wouldn't start up :) – dyasny Nov 22 '11 at 10:52
I have same issue. I have ubuntu installed and i want to install centos inside ubuntu. I have successfully installed but when i am going to reboot it gets no bootable device error. can i write ONBOOT=yes in ubuntu eth0. – Suresh Jan 23 '13 at 06:07
I had this same issue and my fix was changing BOOTPROTO from `none` to `static` (i was assigning an ip address manually) – castis Jun 10 '15 at 21:55

score 0 · Answer 2 · answered Nov 21 '11 at 14:47

0

OK, the settings seem ok. This leaves
1. try to disable iptables completely 2. switch the network device from RTL to virtio

Here's how I have IPTables set up normally: http://fpaste.org/OKG5/

answered Nov 21 '11 at 14:47

dyasny

18,482
6
48
63

Thanks very much for your feedback. Here's how it went: 1. I've disabled iptables by running _service iptables stop_ and then restarted the network with _service network restart_ and confirmed that iptables was disabled with _iptables -L_, which is showing ACCEPT across the board. I restarted the guest and tried to access the network and received the same problems. Also note that I had to run _ifup eth0_ to actually get the adapter to come up. – a.pollock Nov 21 '11 at 15:11
2. I wasn't entirely sure how to do this. A quick look at the libvirt website led me to add to the domain XML under the interfaces element. After that I reran the guest, but still I'm having the same issue. If I've mistaken what you meant by switching the network device from RTL to virtio, do please let me know. I really appreciate the help you've given so far! – a.pollock Nov 21 '11 at 15:13
does the VM actually get an IP from DHCP? – dyasny Nov 21 '11 at 15:41
It doesn't appear to, no. _eth0_ doesn't seem to be up initially. I have to run _ifup eth0_ to get it up, and even then it doesn't appear to have an IPv4 address (it does seem to have an IPv6 address though). I've updated the question with the printout from _ifconfig -a_ before and after bringing up _eth0_. – a.pollock Nov 21 '11 at 15:45
I checked the logs on the DHCP server and it is not receiving any DHCP requests from the guest OS. – a.pollock Nov 21 '11 at 15:53
http://fpaste.org/WW3k/ is what my VM looks like. http://fpaste.org/Gnjo/ is yours. To compare to me xml: http://fpaste.org/inDW/ – dyasny Nov 21 '11 at 15:59
There are some differences, but I'm afraid I'm not aware of what these differences would be affecting. For example, on the -netdev option, I'm using fd 20 and you're using fd 22, but this could be due to differences between Linux and Windows guest OSs? – a.pollock Nov 21 '11 at 16:12
yeah, those aren't relevant really. I can see two places where things might have gone wrong - in the guest itself, assuming the network is set up correctly, and in the connection between the bridge and the tap device the VM is using. At this point, I would start trying to ping an IP from the VM, and running tcpdump on the host to see where things go wrong. Might be that the VM is reaching out through the bridge, but the switch is preventing multiple MAC sources from a single port (known security measure) – dyasny Nov 21 '11 at 16:25
I wouldn't know where to begin with knowing what would be wrong with the network set up on the guest. I assume kvm "injected" the network stuff when starting the VM. As for the tcpdump, I received the following messages when the VM first started up: [link](http://pastebin.com/Np5ZVDHx). There was no indication, when attempting to ping from the guest, that it was reaching the host's adapter at all. – a.pollock Nov 21 '11 at 16:53
start tcpdump on the host, listening to the bridge interface, and kick the pings off. Then see if the VM reaches the bridge at all – dyasny Nov 21 '11 at 17:28
@a.pollock: eth0 isn't supposed to look like it's up. That's the bridge's job using the hardware of the eth0 interface. – Magellan Nov 21 '11 at 20:57
@AdrianK Is that on the host or the guest? The guest was only displaying a local loopback adapter, and had a eth0 adapter registered but set to not come up on boot. – a.pollock Nov 21 '11 at 21:09
@dyasny There was nothing showing in the tcpdump to suggest that it was even getting to the bridge interface. The ping never actually happens, I just get the "connect: Network is unreachable" message. – a.pollock Nov 21 '11 at 21:25
@dyasny I've managed to work out what the issue was and have added an answer to this question. It was pretty much PEBKAC. I would really like to thank you for your help though! – a.pollock Nov 22 '11 at 10:48
@a.pollock On the host. The guest shouldn't see anything but virtual hardware unless you're doing something fancy with PCI passthrough. I haven't worked with that yet though. – Magellan Nov 22 '11 at 17:51

qemu-kvm/virsh: No network connectivity whilst using bridged networking

2 Answers2