I’m giving a short talk/demo on dogtag pki in ‘Hardware and Cryptography Devroom’ at FOSDEM conf. Here are the details.
Looking forward to the already packed schedule.
Monthly Archives: January 2012
Heya FOSDEM
Filed under Uncategorized
Nested Virtualization with KVM and AMD
After my previous attempt the other day to create a nested-guest(kvm on kvm) with Intel arch, I got hold of an AMD server machine with virt-extensions enabled and gave it a whirl. This went slightly smoother than the Intel attempt.
Some config info about the physical host, regular-guest and nested-guest. (All of them are Fedora-16; x86_64)
- Physical Host (Host hypervisor/Bare metal)
-
[root@phy-host-amd]# virsh nodeinfo CPU model: x86_64 CPU(s): 16 CPU frequency: 2000 MHz CPU socket(s): 2 Core(s) per socket: 8 Thread(s) per core: 1 NUMA cell(s): 1 Memory size: 8173352 kB
- Regualr Guest (Or Guest Hypervisor)
- Config: 4GB Memory; 6 vcpus; 22GB Raw disk image w/ cache=’none’ enabled in the libvirt xml
- Nested Guest
- Config: 2GB Memory; 3 vcpus; 10G Raw disk image
Ensure nesting is enabled on the physical host
Let’s ensure kvm_amd kernel module is enabled with ‘nested’ virt.
[root@phy-host-amd ~]# modinfo kvm_amd | grep -i nested
parm: nested:int
[root@phy-host-amd ~]#
[root@phy-host-amd ~]# cat /sys/module/kvm_amd/parameters/nested
1
[root@phy-host-amd ~]#
[root@phy-host-amd ~]# systool -m kvm_amd -v | grep -i nested
nested = "1"
[root@phy-host-amd ~]#
CAVEAT: To make life a little easier, I configured bridged networking on the physical host to ensure our regular-guest gets a bridged IP; and later, nested-guest gets a NATed IP. I’m noting it here because, the physical host initially had no bridging. The default libvirt bridge virbr0 has 192.168.122.0/24 IP space. So once we set up the regular-guest(or guest-hypervisor), we’ll end up having the same IP space. I tried to fix this prob. by creating another ‘persistent’ libvirt network interface and enabled autostart of it. [virsh net-add; virsh net-define; virsh net-autostart ]. But, it wasn’t elegant and messed up networks on reboot.
Set up the guest hypervisor
Create a minimal regular-guest using virt-install . The one I used is posted here
Now, add the cpu attribute to the regular-guest’s libvirt xml to expose AMD’s svm instructions, which comes with Opteron_G3 model .
Edit the xml using virsh:
# virsh edit regualr-guest
(which will also define the xml)
Here is the attribute to be added to the guest hypervisor’s libvirt xml:
<cpu>
<arch>x86_64</arch>
<model>Opteron_G3</model>
<vendor>AMD</vendor>
<topology sockets='2' cores='8' threads='1'/>
<feature name='wdt'/>
<feature name='skinit'/>
<feature name='osvw'/>
<feature name='3dnowprefetch'/>
<feature name='cr8legacy'/>
<feature name='extapic'/>
<feature name='cmp_legacy'/>
<feature name='3dnow'/>
<feature name='3dnowext'/>
<feature name='pdpe1gb'/>
<feature name='fxsr_opt'/>
<feature name='mmxext'/>
<feature name='ht'/>
<feature name='vme'/>
</cpu>
And, restarted the regular-guest, so that it boots w/ the -cpuflag which the AMD virt extensions:
[root@phy-host-amd ~]# ps -ef | grep -i qemu-kvm qemu 26677 1 14 10:39 ? 00:00:30 /usr/bin/qemu-kvm -S -M pc-0.14 -cpu phenom,+wdt,+skinit,+osvw,+3dnowprefetch,+misalignsse,+sse4a,+abm,+cr8legacy,+extapic,+cmp_legacy,+lahf_lm,+rdtscp,+pdpe1gb,+popcnt,+cx16,+ht,+vme -enable-kvm -m 4096 -smp 6,sockets=2,cores=8,threads=1 -name regular-guest -uuid 8f6a4478-496b-51d8-2de2-ff7fdb964af3 -nographic -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/regular-guest.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -drive file=/var/lib/libvirt/images/regular-guest.img,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=24,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:5f:c6:5f,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
Now, let’s fetch the IP of the regular-guest using virt-cat
[root@phy-host-amd ~]# virsh list Id Name State ---------------------------------- 5 regular-guest running [root@phy-host-amd ~]# [root@phy-host-amd ~]# virt-cat regular-guest /var/log/messages | grep 'dhclient.*bound to' Jan 17 10:13:06 dhcpyy-zz dhclient[732]: bound to ww.xx.yy.zz -- renewal in 32578 seconds.
(Note: ‘ww.xx.yy.zz’ above will be a bridged IP address)
Create the nested guest
Now. install virt-packages in the regular-guest. Also, let’s check if the /dev/kvm char device is exposed in the regular-guest ; and start the libvirtd service.
[root@regular-guest ~]# file /dev/kvm
/dev/kvm: character special
[root@regular-guest ~]# systemctl status libvirtd.service
libvirtd.service - LSB: daemon for libvirt virtualization API
Loaded: loaded (/etc/rc.d/init.d/libvirtd)
Active: active (running) since Tue, 17 Jan 2012 10:49:25 -0500; 5s ago
Process: 1440 ExecStart=/etc/rc.d/init.d/libvirtd start (code=exited, status=0/SUCCESS)
Main PID: 1448 (libvirtd)
CGroup: name=systemd:/system/libvirtd.service
├ 1448 libvirtd --daemon
└ 1501 /usr/sbin/dnsmasq --strict-order --bind-interfaces --pid-file=/var/run/libvirt/network/default.pid --conf-file= --exce...
Proceed with installing a minimal F16 nested-guest w/ virt-install. The script I used is here
Debugging note: Once the guest install is finished, fix the serial console access by disabling plymouth-service using this workaround. This will let us login via virsh serial console(to get kernel and boot messages) w/o any line breaks while entering credentials:
# ln -s /dev/null /etc/systemd/system/plymouth-start.service
Get the (NATed) IP of the nested-guest. (Also, grepped for the qemu-kvm command-line of the nested-guest.)
[root@regular-guest ~]# virsh list Id Name State ---------------------------------- 2 nested-guest running [root@regular-guest ~]# ps -ef | grep qemu-kvm qemu 2245 1 2 Jan17 ? 00:20:11 /usr/bin/qemu-kvm -S -M pc-0.14 -enable-kvm -m 2048 -smp 3,sockets=3,cores=1,threads=1 -name nested-guest -uuid 2aae2ab5-ddb6-2585-aa16-7fe97296f34b -nographic -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/nested-guest.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -drive file=/var/lib/libvirt/images/nested-guest.img,if=none,id=drive-virtio-disk0,format=raw -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=24,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:0e:4e:53,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 [root@regular-guest ~]# virt-cat nested-guest /var/log/messages | grep 'dhclient.*bound to' Jan 17 11:08:30 localhost dhclient[721]: bound to 192.168.122.220 -- renewal in 1393 seconds. [root@regular-guest ~]#
SSh into the nested-guest, install virt-what package and run to see if we’re on a hypervisor
[root@localhost ~]# cat /etc/fedora-release
Fedora release 16 (Verne)
[root@localhost ~]# ifconfig eth0 | grep inet
inet addr:192.168.122.220 Bcast:192.168.122.255 Mask:255.255.255.0
inet6 addr: fe80::5054:ff:fe0e:4e53/64 Scope:Link
[root@localhost ~]#
[root@localhost ~]# virt-what
kvm
Wooo!! so we’re on an OS which is inside an OS which is inside an OS.
Filed under Uncategorized
Nested Virtualization with KVM Intel
Some context: In regular virtualization, your physical linux host is the hypervisor, and runs multiple operating systems. Nested Virtualization let’s you run a guest inside a regular guest(essentially a Guest hypervisor).For AMD there is nested-support available since a while, and some people reported success w/ nesting KVM guests. For Intel arch., there is support available recently, an year-ish, and some in progress work, so thought I’d give it a whirl when Adam Young started discussion about it in context of openstack project.
Some of the common use-cases for that are being discussed for nested-virtualization
– For instance, a cloud user gets a beefy, Regualar Guest(which she completely controls). Now, this user can turn regular guest into a hypervisor, and can cheerfully run/manage multiple guests for developing or testing w/o the hassle and intervention of the cloud provider.
– Possibility of having a many instances of virtualization setup (hypervisor and its guests) on one single Bare metal.
– Ability to debug and test hypervisor software
I have immediate access to a moderately beefy Intel hardware, and rest of the post is based on Intel’s CPU virt extensions. Before proceeding, let’s settle on some terminology for clarity:
- Physical Host (Host hypervisor/Bare metal)
- Config: Intel(R) Xeon(R) CPU(4 cores/socket); 10GB Memory; CPU Freq – 2GHz; Running latest Fedora-16(Minimal foot-print, @core only with Virt pkgs;x86_64; kernel-3.1.8-2.fc16.x86_64
- Regualr Guest (Or Guest Hypervisor)
- Config: 4GB Memory; 4vCPU; 20GB Raw disk image with cache =’none’ to have decent I/O; Minimal, @core F16; And same virt-packages as Physical Host; x86_64
- Nested Guest (Guest installed inside the Regular Guest)
- Config: 2GB Memory; 1vCPU; Minimal(@core only) F16; x86_64
Enabling Nesting on the Physical Host
Node Info of the Physical Host.
# virsh nodeinfo CPU model: x86_64 CPU(s): 4 CPU frequency: 1994 MHz CPU socket(s): 1 Core(s) per socket: 4 Thread(s) per core: 1 NUMA cell(s): 1 Memory size: 10242864 kB
Let us first ensure kvm_intel kernel module has nesting enabled. By default, it’s disabled for Intel arch[ but enabled for AMD — SVM (secure virtual machine) extensions arch.]
# modinfo kvm_intel | grep -i nested parm: nested:bool #
And, we need to pass this kvm-intel.nested=1 on kernel commandline while rebooting the host to enable nesting for the Intel KVM kernel module. Which can be verified after boot by doing:
# cat /sys/module/kvm_intel/parameters/nested
Y
# systool -m kvm_intel -v | grep -i nested
nested = "Y"
#
Or alternatively, Adam Young identified that nesting can be enabled by adding this directive options kvm-intel nested=y to the end of /etc/modprobe.d/dist.conf file and reboot the host so it persists.
Set up the Regular Guest(or Guest hypervisor)
Install a regular guest using virt-install or oz tool or any other preferred way. I made a quick script here. And ensure to have cache=’none’ in the disk attribute of the Guest Hypervisor’s xml file. (observation: Install via virt-install tool didn’t seem have this option picked by default.) Here is the ‘drive’ attribute libvirt xml snippet:
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source file='/var/lib/libvirt/images/regular-guest.img'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
Now, let’s try to enable Intel VMX(Virtual Machine Extensions) in the regular guest’s CPU. We can do it by running the below on the Physical host(aka Host Hypervisor), and adding the ‘cpu’ attribute to the regular-guest’s libvirt xml file, and start the guest.
# virsh capabilities | virsh cpu-baseline /dev/stdin <cpu match='exact'> <model>Penryn</model> <vendor>Intel</vendor> <feature policy='require' name='dca'/> <feature policy='require' name='xtpr'/> <feature policy='require' name='tm2'/> <feature policy='require' name='vmx'/> <feature policy='require' name='ds_cpl'/> <feature policy='require' name='monitor'/> <feature policy='require' name='pbe'/> <feature policy='require' name='tm'/> <feature policy='require' name='ht'/> <feature policy='require' name='ss'/> <feature policy='require' name='acpi'/> <feature policy='require' name='ds'/> <feature policy='require' name='vme'/> </cpu>
The o/p of the above cmd has a variety of options. Since we need only vmx extensions, I tried the simple way by adding to the regular-guest’s libvirt xml(virsh edit ..) and started it.
<cpu match='exact'> <model>core2duo</model> <feature policy='require' name='vmx'/> </cpu>
Thanks to Jiri Denemark for the above hint. Also note that, there is a very detailed and informative post from Dan P Berrange on host/guest CPU models in libvirt.
As we enabled vmx in the guest-hypervisor, let’s confirm that vmx is exposed in the emulated CPU by ensuring qemu-kvm is invoked with -cpu core2duo,+vmx :
[root@physical-host ~]# ps -ef | grep qemu-kvm qemu 17102 1 4 22:29 ? 00:00:34 /usr/bin/qemu-kvm -S -M pc-0.14 -cpu core2duo,+vmx -enable-kvm -m 3072 -smp 3,sockets=3,cores=1,threads=1 -name f16test1 -uuid f6219dbd-f515-f3c8-a7e8-832b99a24b5d -nographic -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/f16test1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -drive file=/export/vmimgs/f16test1.img,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=21,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:e6:cc:4e,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
Now, let’s attempt to create a nested guest
Here comes the more interesting part, the nested-guest config. will be 2G RAM; 1vcpu; 8GB virtual disk. And let’s invoke a virt-install cmdline with a minimal kickstart install:
[root@regular-guest ~]# virt-install --connect=qemu:///system \
--network=bridge:virbr0 \
--initrd-inject=/root/fed.ks \
--extra-args=ks=file:/fed.ks console=tty0 console=ttyS0,115200 serial rd_NO_PLYMOUTH \
--name=nested-guest --disk path=/var/lib/libvirt/images/nested-guest.img,size=6 \
--ram 2048 \
--vcpus=1 \
--check-cpu \
--hvm \
--location=http://download.foo.bar.com/pub/fedora/linux/releases/16/Fedora/x86_64/os/
--nographics
Starting install...
Retrieving file .treeinfo... | 1.7 kB 00:00 ...
Retrieving file vmlinuz... | 7.9 MB 00:08 ...
Retrieving file initrd.img... 28% [============== ] 647 kB/s | 38 MB 02:25 ETA
virt-install proceeds fine(to a certain extent), doing all regular things like getting access to network, create devices, create file-systems, dep checks performed, and finally package install proceeds:
Welcome to Fedora for x86_64
┌─────────────────────┤ Package Installation ├──────────────────────┐
│ │
│ │
│ 24% │
│ │
│ Packages completed: 52 of 390 │
│ │
│ Installing glibc-common-2.14.90-14.x86_64 (112 MB) │
│ Common binaries and locale data for glibc │
│ │
│ │
│ │
└───────────────────────────────────────────────────────────────────┘
And now, it’s stuck like that for ever. Doesn’t budge, trying to install pkgs for eternity. Let’s try to see what’s the state of the guest in a seperate terminal
[root@regular-guest ~]# virsh list Id Name State ---------------------------------- 1 nested-guest paused [root@regular-guest ~]# [root@regular-guest ~]# virsh domstate nested-guest --reason paused (unknown) [root@regular-guest ~]#
So our nested-guest seems to be paused, And package install on the nested-guest’s serial console is still hung. I gave up at this point. Need to try if I can get any helpful info w/ virt-dmesg tool aor any other ways to debug this further.
Just to note, there is enough disk space and memory on the ‘regular-guest’, so that case is ruled out here. And, I tried to destroy the broken nested-guest, and attempted to create a fresh one(repeated twice). Still no dice.
So not much luck yet with Intel arch, I’d have to try on an AMD machine.
UPDATE(on Intel arch): After trying a couple of times, I was finally able to ssh to the nested guest, but, after a reboot, the nested-guest loses the IP rendering it inaccessible.(Info: the regular-guest has a bridged IP, and nested-guest has a NATed IP) . And I couldn’t login via serial-console, as it’s broken due to a regression(which has a workaround). Also, refer to comments below for further discussion on NATed networking caveats.
UPDATE2: The correct syntax to be added to /etc/modprobe.conf/dist.conf is options kvm-intel nested=y
Filed under Uncategorized
Revisiting Native Linux KVM Tool
Just a quick revisit of native linux kvm tool(nlkt). There were quite a few improvements upstream. So, I git pulled the latest, built kernel; built the binary executable. The nlkt binary is now renamed to ‘lkvm’ (thanks Pekka, it’ll improve searchability a lot)
Some enhancements I noticed from my testing:
– 9pfs enhancements
– Writable support for qcow2 disk-images
– sandbox support — this seems to be mostly a wrapper around ‘run’ command
After building, I posted latest kvm tool binary lkvm, kernel bzImage, linux .config and init binaries over here . Also, a couple of simple test results with latest git.
To try out a slightly long way, clone the nlkt git tree, (also ensure to have the correct directives enabled in the linux config. I posted mine above) ; build the kernel and kvm tool.
Build:
# cd linux-kvm # make -j5 # cd tools/kvm # make
To give a quick try with the binaries I posted above, first let’s setup default rootfs by running the setup command. Note that we also need to have a guest directory with init and init_stage2 binaries. Where the init mounts the host file system as read-only, runs the init_stage2 to setup a tty console and call the shell executable /bin/sh
-------------------------------------------- [kashyap@tesla nlkt-jan11]$ #./lkvm setup default -------------------------------------------- [kashyap@tesla nlkt-jan11]$ pwd /var/tmp/nlkt-jan11 -------------------------------------------- [kashyap@tesla nlkt-jan11]$ tree . ├── bzImage ├── guest │ ├── init │ └── init_stage2 └── lkvm 1 directory, 4 files [kashyap@tesla nlkt-jan11]$
Once we boot into our default rootfs setup, let’s boot into the kernel
[kashyap@tesla nlkt-jan11]$ ./lkvm run -d default # lkvm run -k ./bzImage -m 448 -c 4 --name default . . . Starting '/bin/sh'... sh-4.2#
We can also notice the host file system being mounted read-only in the guest:
-------- sh-4.2# pwd / -------- sh-4.2# ls bin etc host lib64 root sys usr virt dev home lib proc sbin tmp var -------- sh-4.2# ls host/ ; cd host bin dev home lib64 media opt root sbin sys usr boot etc lib lost+found mnt proc run srv tmp var -------- sh-4.2# touch foo touch: cannot touch `foo': Read-only file system sh-4.2# --------
Now, let’s try the sandbox, which will run a command as part of the init and then exits gracefully . In this case, it’s a simple ls command.
-------- [kashyap@tesla nlkt-jan11]$ ./lkvm sandbox -k ./bzImage -- ls # lkvm run -k ./bzImage -m 448 -c 4 --name guest-9990 . . . Mounting... Starting '/bin/sh'... bin etc host lib64 root sys usr virt dev home lib proc sbin tmp var [ 2.052463] Unregister pv shared memory for cpu 1 [ 2.052546] Unregister pv shared memory for cpu 0 [ 2.052578] Unregister pv shared memory for cpu 3 [ 2.055887] Unregister pv shared memory for cpu 2 [ 2.057093] Restarting system. [ 2.057407] machine restart # KVM session ended normally. [kashyap@tesla nlkt-jan11]$ --------
NOTE: I just cleared some of the stdout for brevity.
UPDATE: Pekka Enberg reminded me in a comment below that I missed to note two more additional user-visible features — PPC64 architecture support ; Serial console emulation is much more faster. (I totally agree there.)
Filed under Uncategorized
Kashyap Chamarthy 