how to enable L3 cache on kvm guest?

Question

I start a kvm guest(centos6.5) on a Intel machine(centos6.5),use libvirt, the guest's xml as follow

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>test-1</name>
  <uuid>9377bce1-ae83-e356-ed15-919c8625fb4b</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static' current='2'>8</vcpu>
  <os>
    <type arch='x86_64' machine='rhel6.5.0'>hvm</type>
    <boot dev='hd'/>
    <boot dev='cdrom'/>
    <bootmenu enable='yes'/>
    <bios useserial='yes' rebootTimeout='0'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='host-passthrough'>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/data/vhosts//test-1.disk'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:ea:12:d9'/>
      <source bridge='br-ex'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:env name='SPICE_DEBUG_ALLOW_MC' value='1'/>
  </qemu:commandline>
</domain>

Now I am confused,Despite the use of "host-passthrough",I still can't see L3 cache in guest,only have L1,L2 cache,as follow

[root@vm-kvm-115 results]# lscpu 
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                2
On-line CPU(s) list:   0,1
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             2
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 26
Stepping:              5
CPU MHz:               2266.746
BogoMIPS:              4533.49
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              4096K
NUMA node0 CPU(s):     0,1

Following are my physical machine information

[root@host-kvm-22 linux]# rpm -qa | grep libvirt
libvirt-client-0.10.2-54.el6_7.2.x86_64
libvirt-0.10.2-54.el6_7.2.x86_64
libvirt-devel-0.10.2-54.el6_7.2.x86_64
libvirt-python-0.10.2-54.el6_7.2.x86_64

[root@host-kvm-22 linux]# rpm -qa | grep qemu
qemu-img-0.12.1.2-2.479.el6_7.2.x86_64
gpxe-roms-qemu-0.9.7-6.14.el6.noarch
qemu-kvm-0.12.1.2-2.479.el6_7.2.x86_64

[root@host-kvm-22 linux]# uname -r
2.6.32-573.8.1.el6.x86_64

[root@host-kvm-22 linux]# lscpu 
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                16
On-line CPU(s) list:   0-15
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 26
Stepping:              5
CPU MHz:               2266.743
BogoMIPS:              4532.68
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              8192K
NUMA node0 CPU(s):     0-3,8-11
NUMA node1 CPU(s):     4-7,12-15

Can anyone tell me how to enable the L3 cache on guest?

Yes I'm agree with Michael... it's used by a single core and your vps inherits it. — Francesco P, Nov 20 '15 at 08:26
As you say,my vps inherits it,then it should also inherit L3 cache,Is not it? — jython.li, Nov 20 '15 at 08:56
It does matters. Just because it wasn't exposed back then doesn't mean it didn't have a perf impact all that time. — Florian Heigl, Nov 12 '20 at 18:11

score 4 · Accepted Answer · answered Dec 15 '16 at 10:15

FWIW, you're misunderstanding the scope of the host-passthrough CPU model. It only controls identification of the CPU and its feature flags - some aspects of the CPU are still not exposed to the guest. For example with the XML you have there, all 8 CPUs are exposed to the guest as separate sockets in the same NUMA node. Your host instead have 2 NUMA nodes, each with two sockets, each with four cores. That alone is probably enough to make the concept of L3 cache in the host not map well into the guest.

You can setup the virtual CPU topology in the XML, but I still don't think it'd make the L3 cache appear. It also doesn't really matter, because you're allowing 8 virtual CPUs to float across all 16 host CPUs. Since your host CPUs spread across 2 NUMA nodes, you're going to be getting cross-NUMA node memory access much of time which has a high latency penalty, which will wipe out any benefit of the cache. IOW, you'd be far better to focus on more efficient VM placement, by using CPU pinning to confine the guest to a single host NUMA node.

score 2 · Answer 2 · answered Jul 18 '19 at 12:00

2

L3 cache support is added to QEMU from version 2.8.0. See Bugzilla.

Exposing L3 cache to guest will improve performance as guest CPUs can avoid a lot of IPI (Inter-Processor Interrupt). Please read this article for more information.

answered Jul 18 '19 at 12:00

theB

161
2

how to enable L3 cache on kvm guest?

2 Answers2