Experiment with ‘Native Linux KVM Tool’

Over the weekend I was tinkering around with the recently announced version 2 of ‘Native Linux KVM tool’. My aim was to boot a minimal Linux (RAW) disk image and a QCOW2 based image. Read on for details:

A little bit of context
‘Native Linux KVM tool’ was first announced by Pekka Enberg on lkml/kvm-upstream lists. From the announce email:

“Right now it can boot a Linux image and provide you output via a serial
console, over the host terminal, i.e. you can use it to boot a guest
Linux image in a terminal or over ssh and log into the guest without
much guest or host side setup work needed.”

Essentially, the initial goal of the tool appears to be a featherlight-weight userspace alternative to QEMU, which can boot Linux guests. And this Native tool lives inside the kernel tree under /tools(this means, if/when this tool is merged in the mainline kernel tree, a linux distro will by default get minimal userspace tool to boot linux guests). However, QEMU does plenty more than booting a linux guest.(Quick googling will provide all the info.)

Getting to the matter
On Saturday, I started off by pulling Penberg’s kernel git tree[1]. And proceeded to configure it with some build time options required for ‘Native Linux KVM tool’ by following the Version 2 announcement. And I enabled each of the necessary options in the kernel configuration. Before I proceed, Sasha Levin(Native KVM tool dev.) suggested that I compile the guest kernel without any modules. Reason being, if I built it with modules, all of them needs to be explicitly loaded into the disk image before booting. If we have all of them built in, we just use the ‘bzImage’. So, I did a quick `sed -i ‘s/=m/=y/’ .config` to *include* everything and compiled the kernel with ‘make -j5’ (so that it builds with 5 parallel threads).

Build complete, bzImage generated. Without any delay I went ahead and launched the hypervisor to boot the minimal disk image:

  [root@moon kvm]#./kvm run -d /export/test-images/linux-0.2.img 
 # kvm run -k ../../arch/x86/boot/bzImage -m 448 -c 4 --name guest-9408
.
.
.
[   63.903734] Copyright (c) 2009 - 2010 Intel Corporation.
[   63.903753] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version
3.3.8-k2
[   63.903755] ixgbe: Copyright (c) 1999-2011 Intel Corporation.
[   63.903774] ixgbevf: Intel(R) 10 Gigabit PCI Exp[root@moon kvm]# 

No dice! As it can be noticed, it just exits abruptly w/o providing any useful error(And later I was pointed out that it didn’t go through the regular exit path either). So I provided my kernel .config and bzImage for debugging.

Meanwhile, Sasha provided me his kernel .config file to get a working ‘kvm’ tool. Rinse and repeat – re-compilation and booting the disk image.

Woot! this time it boots into the minimal linux disk image.

 [root@moon kvm]# ./kvm run -d /export/test-images/linux-0.2.img
 # kvm run -k ../../arch/x86/boot/bzImage -m 448 -c 4 --name guest-9408
.
.
.
[    1.386747] Bluetooth: Virtual HCI driver ver 1.3
[    1.387838] Bluetooth: HCI UART driver ver 2.2
[    1.388615] Bluetooth: HCI H4 protocol initialized
[    1.389481] Bluetooth: HCI BCSP protocol initialized
[    1.390366] Bluetooth: HCILL protocol initialized
sh-2.05b# mkdir /export 
sh-2.05b# pwd 
/export
sh-2.05b# echo test > foo.txt
sh-2.05b# cat foo.txt 
test
sh-2.05b# ls /
bin   dev  export  lost+found  proc  sbin  usr
boot  etc  lib	   mnt	       root  tmp   var
sh-2.05b# df -hT
Filesystem    Type    Size  Used Avail Use% Mounted on
rootfs      rootfs     20M   17M  1.7M  92% /
/dev/root     ext4     20M   17M  1.7M  92% /
devtmpfs  devtmpfs    150M     0  150M   0% /dev
sh-2.05b# 

Next morning, on Sunday, I tried to do a couple of regular virtual guest operations like pause, resume, list, stop. I first chose to ‘pause’ the guest. Poof! I’m thrown out of my ssh session to the *host* , where I’m doing all of this. I can’ re-ssh any more, the SSH session was killed. All I see is a connection refused. Confused, I checked with Penberg, he replied, it’s a nasty bug(which sends the pause signal to the wrong process), but fixed and pointed me to the git commit. Ok, that confirmed I didn’t screw up anything. I re-pulled and re-compiled kernel tree on my Lenovo X200 laptop . (Later in the day, I went and fixed the host test machine(this reminded me to tie this machine to a remote power management console).

With the up2date git tree (and post recompilation), I was able to do all the regular virtual guest operations fine.

Results
Test env.
OS: Fedora -15
Processor: 4 Intel(R) Xeon(R) CPU E5405 @ 2.00GHz
RAM: 8GB

I tested w/ both minimal RAW and a debian squeeze (de-compressed) QCOW2 disk images.

With minimal RAW image: I was able to run; pause; resume; list; stop the minimal RAW linux disk image. Also my (limited)test results, guest networking is a little fragile(understandably) at the moment.
Posted my results here.

With QCOW2 image: Currently a QCOW2 image boots in ‘read-only’ mode. To try it, below is the working syntax.

# ./kvm run -p "root=/dev/vda1" -d /export/test-images/debian_squeeze_i386_standard_decompressed.qcow2

For more verbose details, errors I encountered, conversion of compressed to de-compressed images, I posted my notes and results here.

NOTE: I’m yet to boot any of the Fedora/RHEL RAW disk images which I use in my daily work. Also, I haven’t tried out the ‘virtio ballon’ feature. Will write more on these when I get to it.

For a quick shot
If you don’t want to compile your own kernel and do all the stuff, to give a quick shot, all that needs to be done is:

Download the working Kernel x86 boot executable ‘bzImage’(I built with Sasha’s config); compiled ‘kvm’ tool binary ; and a minimal linux disk image(linux-0.2.img)
– ‘bzImage’ and ‘kvm’ tool binary are located on my fedora people page
– minimal linux disk image — # wget http://wiki.qemu.org/download/linux-0.2.img.bz2 && bunzip2 linux-0.2.img.bz2

And, run the hypervisor to boot into the minimal disk image:

 
# ./kvm run -k ~/tinker/native-linux-kvm/linux-kvm/arch/x86/boot/bzImage  -d /export/testimages-nlt/linux-0.2.img
 

I jotted down a README here.
If you’re interested to compile your own kernel from scratch w/ latest git, I uploaded a working kernel .config (from Sasha) on my fedora people page.

Thanks a lot to Sasha Levin and Pekka Enberg. They were very helpful answering all my questions and also walked me through some of the issues I was facing.

References

1. git://github.com/penberg/linux-kvm.git
2. http://kashyapc.fedorapeople.org/native-linux-kvm-tool/working-native-linux-kvm-tool/sashal-linux-config

Update:
With this commit 6533f7913743742fdd690eee0930fb7ba1bcbb1f, Pekka introduced an ‘init’ target . So if there are any errors( like /usr/bin/ld: cannot find -lc) while compiling the kvm binary on Fedora, ensure to have glibc-static package installed to get them resolved.

Advertisement