3

I'd like to change IRQ SMP affinity for reasons, outlined in this question: CPU0 is swamped with eth1 interrupts

But I can't — I see Input/output error when I try to write to /proc/irq/*/smp_affinity.

Please point me to the HOWTO on the matter. (A formal reference on /proc/irq/*/ would be cool as well.)

Gory details:

Note that this is a VM (PV domU) inside an Ubuntu-based Xen XCP host.

$ uname -a
Linux MYHOST 2.6.38-15-virtual #59-Ubuntu SMP Fri Apr 27 16:40:18 UTC 2012 i686 i686 i386 GNU/Linux

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 11.04
Release:    11.04
Codename:   natty

$ sudo cat /proc/irq/*/smp_affinity
01
01
01
01
01
80
80
80
80
80
80
40
40
40
40
40
40
20
20
20
20
20
20
10
10
10
10
10
10
08
08
08
08
08
08
04
04
04
04
04
04
02
02
02
02
02
02
01
01
01
01
01
01

Update. The error details:

$ N=$(grep -c processor /proc/cpuinfo)
$ echo $N
8

$ printf %x $((2**N-1))
ff

$ printf %x $((2**N-1)) | sudo tee /proc/irq/*/smp_affinity
fftee: /proc/irq/288/smp_affinity: Input/output error
tee: /proc/irq/289/smp_affinity: Input/output error
tee: /proc/irq/290/smp_affinity: Input/output error
tee: /proc/irq/291/smp_affinity: Input/output error
tee: /proc/irq/292/smp_affinity: Input/output error
tee: /proc/irq/293/smp_affinity: Input/output error
tee: /proc/irq/294/smp_affinity: Input/output error
tee: /proc/irq/295/smp_affinity: Input/output error
tee: /proc/irq/296/smp_affinity: Input/output error
tee: /proc/irq/297/smp_affinity: Input/output error
tee: /proc/irq/298/smp_affinity: Input/output error
tee: /proc/irq/299/smp_affinity: Input/output error
tee: /proc/irq/300/smp_affinity: Input/output error
tee: /proc/irq/301/smp_affinity: Input/output error
tee: /proc/irq/302/smp_affinity: Input/output error
tee: /proc/irq/303/smp_affinity: Input/output error
tee: /proc/irq/304/smp_affinity: Input/output error
tee: /proc/irq/305/smp_affinity: Input/output error
tee: /proc/irq/306/smp_affinity: Input/output error
tee: /proc/irq/307/smp_affinity: Input/output error
tee: /proc/irq/308/smp_affinity: Input/output error
tee: /proc/irq/309/smp_affinity: Input/output error
tee: /proc/irq/310/smp_affinity: Input/output error
tee: /proc/irq/311/smp_affinity: Input/output error
tee: /proc/irq/312/smp_affinity: Input/output error
tee: /proc/irq/313/smp_affinity: Input/output error
tee: /proc/irq/314/smp_affinity: Input/output error
tee: /proc/irq/315/smp_affinity: Input/output error
tee: /proc/irq/316/smp_affinity: Input/output error
tee: /proc/irq/317/smp_affinity: Input/output error
tee: /proc/irq/318/smp_affinity: Input/output error
tee: /proc/irq/319/smp_affinity: Input/output error
tee: /proc/irq/320/smp_affinity: Input/output error
tee: /proc/irq/321/smp_affinity: Input/output error
tee: /proc/irq/322/smp_affinity: Input/output error
tee: /proc/irq/323/smp_affinity: Input/output error
tee: /proc/irq/324/smp_affinity: Input/output error
tee: /proc/irq/325/smp_affinity: Input/output error
tee: /proc/irq/326/smp_affinity: Input/output error
tee: /proc/irq/327/smp_affinity: Input/output error
tee: /proc/irq/328/smp_affinity: Input/output error
tee: /proc/irq/329/smp_affinity: Input/output error
tee: /proc/irq/330/smp_affinity: Input/output error
tee: /proc/irq/331/smp_affinity: Input/output error
tee: /proc/irq/332/smp_affinity: Input/output error
tee: /proc/irq/333/smp_affinity: Input/output error
tee: /proc/irq/334/smp_affinity: Input/output error
tee: /proc/irq/335/smp_affinity: Input/output error

Update. irqbalance is running:

$ sudo service irqbalance status
irqbalance start/running, process 560
Community
  • 1
Alexander Gladysh
  • 2,343
  • 7
  • 30
  • 47

3 Answers3

7

There is a file called Documentation/IRQ-affinity.txt in the Linux source code.

/proc/irq/IRQ#/smp_affinity specifies which target CPUs are permitted
for a given IRQ source. It's a bitmask of allowed CPUs. It's not allowed
to turn off all CPUs, and if an IRQ controller does not support IRQ
affinity then the value will not change from the default 0xffffffff.

The catch here is that the bitmask is in hex. So, if you have N CPUs,

N=$(grep -c processor /proc/cpuinfo)

to enable all IRQs for all CPUs, where you have N CPUs you can

printf %x $((2**N-1)) | sudo tee /proc/irq/*/smp_affinity

Kernel 3.0 and later has a file called smp_affinity_list. This file takes a comma separated list of CPUs or CPU ranges. Valid examples: 0, 2,3,5-7, 0-7. The previous command is equivalent to:

echo 0-$((N-1)) | sudo tee /proc/irq/*/smp_affinity_list
chutz
  • 7,569
  • 1
  • 28
  • 57
3

I do not think that moving interrupts to different CPUs - especially for handling network events - would increase the performance.

The contrary will happen, since the network-code can not be held in a specific CPU any longer.

So as long as you do not experience dropped packets on your network interface, I would say - this is quite normal behaviour for a network that serves many packets.

You need to lower the number of interrupts - moving them around will not help (on the contrary, as I tried to outline).

I see two possible solutions:

  • Raise your MTU-size to lower the number of interrupts
  • PIN the vCPU 0 of the DomU to a dedicated CPU on the Dom0 (that is not used by any other VM or the Dom0).

Update 2012-12-17: Since you asked for authoritative links - I tried to ask a general question When not to use virtualisation - I think this is one of the cases, where you are hitting general VM limits. One of the answers to the question contains a different approach: Use containers, instead of virtualization.

I hope this helps...

Community
  • 1
Nils
  • 7,657
  • 3
  • 31
  • 71
  • 1
    Thanks, but, would it help or not is outside of the scope of this question. Your help is very welcome here: http://serverfault.com/questions/446173/cpu0-is-swamped-with-eth1-interrupts or here: http://serverfault.com/questions/446081/uneven-cpu-core-utilization – Alexander Gladysh Nov 12 '12 at 14:59
  • @AlexanderGladysh Your core question is how to improve performance. Changing affinity (in the DomU) won`t help. I updated my answer accordingly. – Nils Nov 13 '12 at 21:16
  • thank you, I'll look into that. But I'd like to see an authoritative answer on *why* affinity configuration does not work. It worries me that everyone seems to assume that it is possible, but I can't do that. – Alexander Gladysh Nov 13 '12 at 21:59
  • @AlexanderGladysh I doubt that affinity should work on a vCPU in a PV XEN DomU. I even read [somewhere](http://debaan.blogspot.de/2007/04/xen-tips-and-tricks.html) that irq-balance does not make any sense there. – Nils Nov 13 '12 at 22:14
  • I see. Links to the authoritative sources are much appreciated. – Alexander Gladysh Nov 13 '12 at 23:20
  • 1
    @AlexanderGladysh See my updated answer. It is not easy to find in the internet, so I asked here on SF... – Nils Dec 17 '12 at 15:28
0

But I'd like to see an authoritative answer on why affinity configuration does not work. It worries me that everyone seems to assume that it is possible, but I can't do that.

Just on this specific question - and I have no idea whether Xen affects this answer - you get the Input/output error message when no device driver currently has a handler installed for this interrupt.

If a driver installs a handler on '105', for example, then directory /proc/irq/105 is created, and contains an smp_affinity, among other things. You can now write to smp_affinity. If you close the device, the directory remains, with the smp_affinity, but you can no longer write to it. Open the device (via open, or whatever eventually calls request_irq in the driver) and you can now write to the file. Only tested on RHEL/SL 6.

If you don't know whether or not the device is currently open, check the directory. It will contain the device name if it's open.

EML
  • 393
  • 3
  • 12