Revisiting Native Linux KVM Tool

Just a quick revisit of native linux kvm tool(nlkt). There were quite a few improvements upstream. So, I git pulled the latest, built kernel; built the binary executable. The nlkt binary is now renamed to ‘lkvm’ (thanks Pekka, it’ll improve searchability a lot)

Some enhancements I noticed from my testing:
– 9pfs enhancements
– Writable support for qcow2 disk-images
– sandbox support — this seems to be mostly a wrapper around ‘run’ command

After building, I posted latest kvm tool binary lkvm, kernel bzImage, linux .config and init binaries over here . Also, a couple of simple test results with latest git.

To try out a slightly long way, clone the nlkt git tree, (also ensure to have the correct directives enabled in the linux config. I posted mine above) ; build the kernel and kvm tool.

Build:

 
# cd linux-kvm
# make -j5
# cd tools/kvm 
# make
 

To give a quick try with the binaries I posted above, first let’s setup default rootfs by running the setup command. Note that we also need to have a guest directory with init and init_stage2 binaries. Where the init mounts the host file system as read-only, runs the init_stage2 to setup a tty console and call the shell executable /bin/sh

 
--------------------------------------------
[kashyap@tesla nlkt-jan11]$ #./lkvm setup default
--------------------------------------------
[kashyap@tesla nlkt-jan11]$ pwd
/var/tmp/nlkt-jan11
--------------------------------------------
[kashyap@tesla nlkt-jan11]$ tree
.
├── bzImage
├── guest
│   ├── init
│   └── init_stage2
└── lkvm

1 directory, 4 files
[kashyap@tesla nlkt-jan11]$
 

Once we boot into our default rootfs setup, let’s boot into the kernel

 
[kashyap@tesla nlkt-jan11]$ ./lkvm run -d default 
  # lkvm run -k ./bzImage -m 448 -c 4 --name default
.
.
.
Starting '/bin/sh'...
sh-4.2# 
 

We can also notice the host file system being mounted read-only in the guest:

 
--------
sh-4.2# pwd
/
--------
sh-4.2# ls
bin  etc   host  lib64	root  sys  usr	virt
dev  home  lib	 proc	sbin  tmp  var
--------
sh-4.2# ls host/ ; cd host
bin   dev  home  lib64	     media  opt   root	sbin  sys  usr
boot  etc  lib	 lost+found  mnt    proc  run	srv   tmp  var
--------
sh-4.2# touch foo
touch: cannot touch `foo': Read-only file system
sh-4.2# 
--------
 

Now, let’s try the sandbox, which will run a command as part of the init and then exits gracefully . In this case, it’s a simple ls command.

  
--------
[kashyap@tesla nlkt-jan11]$ ./lkvm sandbox -k ./bzImage -- ls
  # lkvm run -k ./bzImage -m 448 -c 4 --name guest-9990
.
.
.
Mounting...
Starting '/bin/sh'...
bin  etc   host  lib64	root  sys  usr	virt
dev  home  lib	 proc	sbin  tmp  var
[    2.052463] Unregister pv shared memory for cpu 1
[    2.052546] Unregister pv shared memory for cpu 0
[    2.052578] Unregister pv shared memory for cpu 3
[    2.055887] Unregister pv shared memory for cpu 2
[    2.057093] Restarting system.
[    2.057407] machine restart

  # KVM session ended normally.
[kashyap@tesla nlkt-jan11]$ 
--------
 

NOTE: I just cleared some of the stdout for brevity.

UPDATE: Pekka Enberg reminded me in a comment below that I missed to note two more additional user-visible features — PPC64 architecture support ; Serial console emulation is much more faster. (I totally agree there.)

Advertisement

Virtio Balloon in action — with native linux kvm tool

Over the weekend, for fun, I decided to tinker with the virtio ballon feature. For simplicity’s sake, I tested it with native linux kvm tool. I previously wrote about it here

If you want to get a quick refresher about virtio balloon concept, a while ago, Richard W Jones made an excellent post w/ nice illustration using images over here

Before I went ahead, I pulled the latest git, and applied a couple of patches from Sasha Levin(nlkt dev), which are on kvm mailing list, which fix the behaviour of kvm tool commands ‘stat’, ‘debug’ and a couple other things. Then I compiled the kvm tool, built the kernel bzImage, on a Fedora 16 box. Here is how it went:

First, setup a new rootfs(which will be stored under ~/.kvm-tools/default), and will mount your host file system in a read-only mode in the guest:

$ ./kvm setup default
  Info: Your new rootfs named default has been created.
You can now start it by running 'kvm run -d default'

$

Then, boot the kernel bzImage with the virtio ballon option. Once, booted, guest shell access over a serial console will be presented.
(Also, the below command line will allocate the guest memory, guest vcpu depending on the host configuration)

$ ./kvm run --balloon -k ../../arch/x86/boot/bzImage                                                                                                                                            
  # kvm run -k ../../arch/x86/boot/bzImage -m 448 -c 4 --name guest-31362
.
.
.
[    1.505724] Bluetooth: BNEP filters: protocol multicast
[    1.508899] Bluetooth: HIDP (Human Interface Emulation) ver 1.2
[    1.511511] lib80211: common routines for IEEE802.11 drivers
[    1.513885] Installing 9P2000 support
[    1.517080] Registering the dns_resolver key type
[    1.518737]   Magic number: 15:85:793
[    1.520495] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
[    1.523497] EDD information not available.
[    1.537035] Sending DHCP requests ., OK
[    1.558038] IP-Config: Got DHCP answer from 192.168.33.1, my address is 192.168.33.15
[    1.560257] IP-Config: Complete:
[    1.561163]      device=eth0, addr=192.168.33.15, mask=255.255.255.0, gw=192.168.33.1,
[    1.563257]      host=192.168.33.15, domain=, nis-domain=(none),
[    1.564861]      bootserver=192.168.33.1, rootserver=0.0.0.0, rootpath=
[    1.567199] VFS: Unable to mount root fs via NFS, trying floppy.
[    1.574154] VFS: Mounted root (9p filesystem) on device 0:13.
[    1.576080] devtmpfs: mounted
[    1.578602] Freeing unused kernel memory: 512k freed
[    1.580356] Write protecting the kernel read-only data: 12288k
[    1.591640] Freeing unused kernel memory: 1916k freed
[    1.601083] Freeing unused kernel memory: 1436k freed
Mounting...
Starting '/bin/sh'...
sh-4.2# 

Get the guest current memory statistics:

$ ./kvm stat -m --all                                                                  


        *** Guest memory statistics ***

The amount of memory that has been swapped in (in bytes):0
The amount of memory that has been swapped out to disk (in bytes):0
The number of major page faults that have occurred:40
The number of minor page faults that have occurred:495
The amount of memory not being used for any purpose (in bytes):438673408
The total amount of memory available (in bytes):449032192

$ 

List the current running guest instances:

$ ./kvm list -i
  PID GUEST
31362 guest-31362
$ 

Now, on a different terminal, let’s try to inflate/deflate the balloon. The current available memory, is roughly 448MB (Refer above. In bytes — 449032192)

To inflate, which means, to return memory from the guest, run the below command, which returns 10MB of memory back to the guest.

$ ./kvm balloon -n guest-31362 --inflate 10
$ 

Run the memory status again. Now, 10485760 bytes(10MB) is reduced from the guest and returned to the host. (which means, the virtio balloon is inflated)

$ ./kvm stat -m --all


        *** Guest memory statistics ***

The amount of memory that has been swapped in (in bytes):0
The amount of memory that has been swapped out to disk (in bytes):0
The number of major page faults that have occurred:40
The number of minor page faults that have occurred:502
The amount of memory not being used for any purpose (in bytes):428339200
The total amount of memory available (in bytes):438546432

$ 

Now, let’s return back the memory to the guest from the host, by running the deflate command

$ ./kvm balloon -n guest-31362 --deflate 10
$ 

Again, re-check the memory status. 10485760 bytes(10MB) is now returned back to the guest from the host. (which means, the virtio balloon is inflated)

$ ./kvm stat -m --all                                                                  


        *** Guest memory statistics ***

The amount of memory that has been swapped in (in bytes):0
The amount of memory that has been swapped out to disk (in bytes):0
The number of major page faults that have occurred:40
The number of minor page faults that have occurred:502
The amount of memory not being used for any purpose (in bytes):436846592
The total amount of memory available (in bytes):449032192

$ 

Now, like me, if you’re wondering what happens if you reduce the memory( inflate) by 10MB, and try to return back(deflate) 100 MB to the guest, it won’t let you.

Thanks to Sasha for helping me to fix some of the compile errors w/ newest gcc.

Experiment with ‘Native Linux KVM Tool’

Over the weekend I was tinkering around with the recently announced version 2 of ‘Native Linux KVM tool’. My aim was to boot a minimal Linux (RAW) disk image and a QCOW2 based image. Read on for details:

A little bit of context
‘Native Linux KVM tool’ was first announced by Pekka Enberg on lkml/kvm-upstream lists. From the announce email:

“Right now it can boot a Linux image and provide you output via a serial
console, over the host terminal, i.e. you can use it to boot a guest
Linux image in a terminal or over ssh and log into the guest without
much guest or host side setup work needed.”

Essentially, the initial goal of the tool appears to be a featherlight-weight userspace alternative to QEMU, which can boot Linux guests. And this Native tool lives inside the kernel tree under /tools(this means, if/when this tool is merged in the mainline kernel tree, a linux distro will by default get minimal userspace tool to boot linux guests). However, QEMU does plenty more than booting a linux guest.(Quick googling will provide all the info.)

Getting to the matter
On Saturday, I started off by pulling Penberg’s kernel git tree[1]. And proceeded to configure it with some build time options required for ‘Native Linux KVM tool’ by following the Version 2 announcement. And I enabled each of the necessary options in the kernel configuration. Before I proceed, Sasha Levin(Native KVM tool dev.) suggested that I compile the guest kernel without any modules. Reason being, if I built it with modules, all of them needs to be explicitly loaded into the disk image before booting. If we have all of them built in, we just use the ‘bzImage’. So, I did a quick `sed -i ‘s/=m/=y/’ .config` to *include* everything and compiled the kernel with ‘make -j5’ (so that it builds with 5 parallel threads).

Build complete, bzImage generated. Without any delay I went ahead and launched the hypervisor to boot the minimal disk image:

  [root@moon kvm]#./kvm run -d /export/test-images/linux-0.2.img 
 # kvm run -k ../../arch/x86/boot/bzImage -m 448 -c 4 --name guest-9408
.
.
.
[   63.903734] Copyright (c) 2009 - 2010 Intel Corporation.
[   63.903753] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version
3.3.8-k2
[   63.903755] ixgbe: Copyright (c) 1999-2011 Intel Corporation.
[   63.903774] ixgbevf: Intel(R) 10 Gigabit PCI Exp[root@moon kvm]# 

No dice! As it can be noticed, it just exits abruptly w/o providing any useful error(And later I was pointed out that it didn’t go through the regular exit path either). So I provided my kernel .config and bzImage for debugging.

Meanwhile, Sasha provided me his kernel .config file to get a working ‘kvm’ tool. Rinse and repeat – re-compilation and booting the disk image.

Woot! this time it boots into the minimal linux disk image.

 [root@moon kvm]# ./kvm run -d /export/test-images/linux-0.2.img
 # kvm run -k ../../arch/x86/boot/bzImage -m 448 -c 4 --name guest-9408
.
.
.
[    1.386747] Bluetooth: Virtual HCI driver ver 1.3
[    1.387838] Bluetooth: HCI UART driver ver 2.2
[    1.388615] Bluetooth: HCI H4 protocol initialized
[    1.389481] Bluetooth: HCI BCSP protocol initialized
[    1.390366] Bluetooth: HCILL protocol initialized
sh-2.05b# mkdir /export 
sh-2.05b# pwd 
/export
sh-2.05b# echo test > foo.txt
sh-2.05b# cat foo.txt 
test
sh-2.05b# ls /
bin   dev  export  lost+found  proc  sbin  usr
boot  etc  lib	   mnt	       root  tmp   var
sh-2.05b# df -hT
Filesystem    Type    Size  Used Avail Use% Mounted on
rootfs      rootfs     20M   17M  1.7M  92% /
/dev/root     ext4     20M   17M  1.7M  92% /
devtmpfs  devtmpfs    150M     0  150M   0% /dev
sh-2.05b# 

Next morning, on Sunday, I tried to do a couple of regular virtual guest operations like pause, resume, list, stop. I first chose to ‘pause’ the guest. Poof! I’m thrown out of my ssh session to the *host* , where I’m doing all of this. I can’ re-ssh any more, the SSH session was killed. All I see is a connection refused. Confused, I checked with Penberg, he replied, it’s a nasty bug(which sends the pause signal to the wrong process), but fixed and pointed me to the git commit. Ok, that confirmed I didn’t screw up anything. I re-pulled and re-compiled kernel tree on my Lenovo X200 laptop . (Later in the day, I went and fixed the host test machine(this reminded me to tie this machine to a remote power management console).

With the up2date git tree (and post recompilation), I was able to do all the regular virtual guest operations fine.

Results
Test env.
OS: Fedora -15
Processor: 4 Intel(R) Xeon(R) CPU E5405 @ 2.00GHz
RAM: 8GB

I tested w/ both minimal RAW and a debian squeeze (de-compressed) QCOW2 disk images.

With minimal RAW image: I was able to run; pause; resume; list; stop the minimal RAW linux disk image. Also my (limited)test results, guest networking is a little fragile(understandably) at the moment.
Posted my results here.

With QCOW2 image: Currently a QCOW2 image boots in ‘read-only’ mode. To try it, below is the working syntax.

# ./kvm run -p "root=/dev/vda1" -d /export/test-images/debian_squeeze_i386_standard_decompressed.qcow2

For more verbose details, errors I encountered, conversion of compressed to de-compressed images, I posted my notes and results here.

NOTE: I’m yet to boot any of the Fedora/RHEL RAW disk images which I use in my daily work. Also, I haven’t tried out the ‘virtio ballon’ feature. Will write more on these when I get to it.

For a quick shot
If you don’t want to compile your own kernel and do all the stuff, to give a quick shot, all that needs to be done is:

Download the working Kernel x86 boot executable ‘bzImage’(I built with Sasha’s config); compiled ‘kvm’ tool binary ; and a minimal linux disk image(linux-0.2.img)
– ‘bzImage’ and ‘kvm’ tool binary are located on my fedora people page
– minimal linux disk image — # wget http://wiki.qemu.org/download/linux-0.2.img.bz2 && bunzip2 linux-0.2.img.bz2

And, run the hypervisor to boot into the minimal disk image:

 
# ./kvm run -k ~/tinker/native-linux-kvm/linux-kvm/arch/x86/boot/bzImage  -d /export/testimages-nlt/linux-0.2.img
 

I jotted down a README here.
If you’re interested to compile your own kernel from scratch w/ latest git, I uploaded a working kernel .config (from Sasha) on my fedora people page.

Thanks a lot to Sasha Levin and Pekka Enberg. They were very helpful answering all my questions and also walked me through some of the issues I was facing.

References

1. git://github.com/penberg/linux-kvm.git
2. http://kashyapc.fedorapeople.org/native-linux-kvm-tool/working-native-linux-kvm-tool/sashal-linux-config

Update:
With this commit 6533f7913743742fdd690eee0930fb7ba1bcbb1f, Pekka introduced an ‘init’ target . So if there are any errors( like /usr/bin/ld: cannot find -lc) while compiling the kvm binary on Fedora, ensure to have glibc-static package installed to get them resolved.