Why I can boot various kerel versions but not XEN kernel in grub2

Question

I have successfully compiled XEN usigng following steps:

wget http://bits.xensource.com/oss-xen/release/4.4.1/xen-4.4.1.tar.gz
tar xzvf xen-4.4.1.tar.gz
cd ./xen-4.4.1
export C_INCLUDE_PATH=/usr/include/x86_64-linux-gnu
./configure
make -j4 dist-xen
make -j4 dist-tools

sudo su
make -j4 install-xen
make -j4 install-tools
echo "GRUB_CMDLINE_XEN_DEFAULT=\"dom0_mem=4096M,max:4096M dom0_max_vcpus=2 dom0_vcpus_pin=true\"" >> /etc/default/grub
echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib" >> ~/.bashrc
update-grub
echo "none /proc/xen xenfs defaults,nofail 0 0" >> /etc/fstab
echo "xen-evtchn" >> /etc/modules
echo "xen-privcmd" >> /etc/modules
update-rc.d xencommons defaults 19 18
update-rc.d xendomains defaults 21 20
update-rc.d xen-watchdog defaults 22 23
reboot

Now my /boot directory looks following:

root@Debian-78-wheezy-64-minimal ~/workplace/backup_xen_boot # ls -la /boot/
total 37M
drwxr-xr-x  4 root root 4.0K Apr 22 14:27 .
drwxr-xr-x 24 root root 4.0K Apr 22 12:47 ..
-rw-r--r--  1 root root 154K Apr  8 19:23 config-3.16.0-0.bpo.4-amd64
-rw-r--r--  1 root root 127K Feb 22 06:16 config-3.2.0-4-amd64
drwxr-xr-x  4 root root  12K Apr 22 15:07 grub
-rw-r--r--  1 root root  15M Apr 22 12:47 initrd.img-3.16.0-0.bpo.4-amd64
-rw-r--r--  1 root root  11M Apr 22 12:38 initrd.img-3.2.0-4-amd64
drwx------  2 root root  16K Apr 22 11:40 lost+found
-rw-r--r--  1 root root 2.6M Apr  8 19:23 System.map-3.16.0-0.bpo.4-amd64
-rw-r--r--  1 root root 2.1M Feb 22 06:16 System.map-3.2.0-4-amd64
-rw-r--r--  1 root root 3.0M Apr  8 19:16 vmlinuz-3.16.0-0.bpo.4-amd64
-rw-r--r--  1 root root 2.8M Feb 22 06:05 vmlinuz-3.2.0-4-amd64
-rw-r--r--  1 root root 831K Apr 22 12:14 xen-4.4.1.gz
lrwxrwxrwx  1 root root   12 Apr 22 12:14 xen-4.4.gz -> xen-4.4.1.gz
lrwxrwxrwx  1 root root   12 Apr 22 12:14 xen-4.gz -> xen-4.4.1.gz

I had to delete: xen.gz which was symbolic link to xen-4.4.1.gz and also xen-syms-4.4.1 (I thing this is debug symbols if not correct me please) due to warning which I get when I've tried update-grub

My grub2 config looks following:

root@Debian-78-wheezy-64-minimal ~/workplace/backup_xen_boot # grep "^menuentry" /boot/grub/grub.cfg | cut -d "'" -f2 | cat -n | awk '{$1=$1-1; print $0}'
0 Debian GNU/Linux, with Linux 3.16.0-0.bpo.4-amd64
1 Debian GNU/Linux, with Linux 3.16.0-0.bpo.4-amd64 (recovery mode)
2 Debian GNU/Linux, with Linux 3.2.0-4-amd64
3 Debian GNU/Linux, with Linux 3.2.0-4-amd64 (recovery mode)
4 Debian GNU/Linux, with Xen 4 and Linux 3.16.0-0.bpo.4-amd64
5 Debian GNU/Linux, with Xen 4 and Linux 3.16.0-0.bpo.4-amd64 (recovery mode)
6 Debian GNU/Linux, with Xen 4 and Linux 3.2.0-4-amd64
7 Debian GNU/Linux, with Xen 4 and Linux 3.2.0-4-amd64 (recovery mode)
8 Debian GNU/Linux, with Xen 4.4 and Linux 3.16.0-0.bpo.4-amd64
9 Debian GNU/Linux, with Xen 4.4 and Linux 3.16.0-0.bpo.4-amd64 (recovery mode)
10 Debian GNU/Linux, with Xen 4.4 and Linux 3.2.0-4-amd64
11 Debian GNU/Linux, with Xen 4.4 and Linux 3.2.0-4-amd64 (recovery mode)
12 Debian GNU/Linux, with Xen 4.4.1 and Linux 3.16.0-0.bpo.4-amd64
13 Debian GNU/Linux, with Xen 4.4.1 and Linux 3.16.0-0.bpo.4-amd64 (recovery mode)
14 Debian GNU/Linux, with Xen 4.4.1 and Linux 3.2.0-4-amd64
15 Debian GNU/Linux, with Xen 4.4.1 and Linux 3.2.0-4-amd64 (recovery mode)

root@Debian-78-wheezy-64-minimal ~/workplace/backup_xen_boot # grep GRUB_DEFAULT /etc/default/grub
GRUB_DEFAULT=12

I am able to boot to kernel 3.2.0-4-amd64 (this is the kernel from which I install XEN and upgraded to 3.16.0-0.bpo.4-amd64) when I set GRUB_DEFAULT=2 and issue update-grub. But the problem is that I cannot boot to XEN kernel even If I have set GRUB_DEFAULT=12. It always boot 3.16.0-0.bpo.4-amd64 kernel as if I had GRUB_DEFAULT=0. Here is proof that I am not using XEN kernel even if i have GRUB_DEFAULT=12 set:

root@Debian-78-wheezy-64-minimal ~/workplace/backup_xen_boot # xen-detect
Not running on Xen.

root@Debian-78-wheezy-64-minimal ~/workplace/backup_xen_boot # uname -a
Linux Debian-78-wheezy-64-minimal 3.16.0-0.bpo.4-amd64 #1 SMP Debian 3.16.7-ckt7-1~bpo70+1 (2015-04-07) x86_64 GNU/Linux

EDIT:

This is also satisfied (even it is related to XEN on RedHat):

before you go to the trouble of building xen, make sure your desired kernel will support dom0. grep XEN /boot/config-3.10* if you don't return at least 35 entries from ^^, then that Redhat kernel does not support dom0.

What else I can try?

Answer 1

All credits goes to guys from #xen channel on freenode, especially: PryMar56 and deHakkelaar

This used to be when it does not works:

ls -la /etc/grub.d/
total 52K
-rwxr-xr-x 1 root root 6.6K Jun  8  2012 00_header
-rwxr-xr-x 1 root root 5.3K Jun  8  2012 05_debian_theme
-rwxr-xr-x 1 root root 5.9K Jan 20  2013 10_linux
-rwxr-xr-x 1 root root 6.2K Jun  8  2012 20_linux_xen
-rwxr-xr-x 1 root root 6.4K Apr 29  2013 30_os-prober
-rwxr-xr-x 1 root root  214 Jun  8  2012 40_custom
-rwxr-xr-x 1 root root   95 Jun  8  2012 41_custom
-rw-r--r-- 1 root root  483 Jun  8  2012 README

And this is when it works:

ls -la /etc/grub.d/
total 52K
-rwxr-xr-x 1 root root 6.6K Jun  8  2012 00_header
-rwxr-xr-x 1 root root 5.3K Jun  8  2012 05_debian_theme
-rwxr-xr-x 1 root root 6.2K Jun  8  2012 09_linux_xen
-rwxr-xr-x 1 root root 5.9K Jan 20  2013 10_linux
-rwxr-xr-x 1 root root 6.4K Apr 29  2013 30_os-prober
-rwxr-xr-x 1 root root  214 Jun  8  2012 40_custom
-rwxr-xr-x 1 root root   95 Jun  8  2012 41_custom
-rw-r--r-- 1 root root  483 Jun  8  2012 README

In summary: xen needs to boot before kernel, So I did:

(cd /etc/grub.d/;mv 20_linux_xen 09_linux_xen)
update-grub

I thing this is something similar