Nested Virtualization — KVM, Intel, with VMCS Shadowing

[Previous installments on Nested Virtualization with KVM and Intel.]

This is part of some recent testing that I’ve been doing with upstream KVM (for 3.10.1). The threads linked here has initial tests bench-marking kernel compile (with make defconfig, a default config file) times in L2. And some minimal guestfish appliance start-up timings in L1.

Some details:

• Setup information to test with VMCS (Virtual Machine Control Structure) Shadowing. In brief, VMCS Shadowing — a processor specific feature — as described upstream, can reduce the overhead of nested virtualization by reducing the number of VMExits from L1 to L0.
• Simple scripts used to create L1 and L2.
• Libvirt XMLs of L1, L2 guests, for reference.

The gritty details of reasons for VMExits are described in Intel architecture manuals, Volume 3b, APPENDIX 1.

Fetching Rawhide Kernel builds using koji

To fetch rawhide kernels from Koji (Fedora build system), this is what I do:

Install the group “RPM Development Tools” to get koji packages

$ yum groupinstall "RPM Development Tools" 

NOTE: If you’re using versions < Fedora 19, please do: yum groupinstall "Development Tools"

Find the latest kernel in Rawhide

$ koji latest-pkg rawhide kernel                                                                      
Build                                     Tag                   Built by
----------------------------------------  --------------------  ----------------
kernel-3.10.0-0.rc0.git26.1.fc20          f20                   jwboyer

Download the resultant kernel from the o/p of the above command.

$ koji download-build \
  --arch=x86_64 kernel-3.10.0-0.rc0.git23.1.fc20 

Finally, install it

$ yum localinstall *.rpm 

Search for a specific patch from an upstream PULL request (KVM) in Fedora Rawhide Kernel

Before I use the latest rawhide kernel for a test, I had to ensure some specific patches made into it from KVM upstream GIT PULL request . This is how I tried:

Fetch the Rawhide Kernel SRPM

Install the rawhide SRPM:

 
# Move into SRPMS directory
$ cd ~/rpmbuild/SRPMS 

# Fetch the SRPM
$ wget http://kojipkgs.fedoraproject.org//packages/kernel/3.10.0/0.rc0.git23.1.fc20/src/kernel-3.10.0-0.rc0.git23.1.fc20.src.rpm 

# Install the kernel SRPM
$ rpm -ivh kernel-3.10.0-0.rc0.git23.1.fc20.src.rpm 

First, simpler way

git-describe to the rescue: Each Fedora rawhide kernel changelog has the precise output of Linus’ tree (thanks jwb!). So, let’s run it on the upstream kernel tree.

 
# Clone Linus' tree (or traverse to its path if it already exists)
# git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
$ cd linux 

# Run:
$ git describe
v3.9-11789-ge0fd9af

The above output can be understood as (from git-describe man page): The current head of the “parent” branch is based on v3.9, and is 11789 commits ahead of it; with a hash suffix ge0fd9af: where “-g” (stands for “git”), e0fd9af is the 7-character abbreviation for the tip commit (which was — e0fd9affeb64088eff407dfc98bbd3a5c17ea479)

Now, compare the above tag from the tag mentioned in changelog of Fedora Rawhide Kernel

$ rpm -q kernel --changelog | head -2 
* Thu May 09 2013 Josh Boyer  - 3.10.0-0.rc0.git23.1
- Linux v3.9-11789-ge0fd9af

Second, a more convoluted way to ensure a specific patch is in

Check upstream KVM tree, for the latest commit –
db6ae6158186a17165ef990bda2895ae7594b039

# Clone the upstream KVM git
$ git clone git://git.kernel.org/pub/scm/virt/kvm/kvm.git

# Traverse into its directory
$ cd kvm 

# Get the latest commit ID
$ git log tags/kvm-3.10-1 | head -1
db6ae6158186a17165ef990bda2895ae7594b039 

Now, get details the above commit. Its commit’s short description says: “kvm: Add compat_ioctl for device control API”

$  git log -p db6ae6158186a17165ef990bda2895ae7594b039 
.....
+       .compat_ioctl = kvm_device_ioctl, 

(that’s the first line of change)

Let’s grep for that string in our just installed kernel sources tree

$ cd ~/rpmbuild/SOURCES/ 

# Copy the patch file archive into a temp directory
$  cp patch-3.9-git23.xz /var/tmp/test/ 

# Extract & list it
$ unxz patch-3.9-git23.xz 
$ ls
patch-3.9-git23 

# grep for that specific line
$ grep ".compat_ioctl = kvm_device_ioctl" patch-3.9-git23 
+       .compat_ioctl = kvm_device_ioctl,

So, the above confirms that the patch (and the KVM GIT PULL request is contained in the koji rawhide build.

And the entire PULL request for KVM updates is merged with this commit ID – 01227a889ed56ae53aeebb9f93be9d54dd8b2de8.

Very late post — KVM Forum 2012 (incoherent) notes

While talking to Amit (thanks!) on IRC, he asked if I ever managed to take notes from the talks I attended at KVM Forum 2012 (side note – that page is up2date with slides from most talks). I did scribble some incoherent notes, but forgot to blog about it. Here we go — http://kashyapc.fedorapeople.org/virt/kvm-forum-2012-notes.txt

Resize a fedora 19 guest with libguestfs tools

The inimitable Rich Jones writes some incredibly useful software. I can’t count how many times they helped me debugging disk images, or gave great insights into different aspects of linux virtualization. One of those instances again — I had to resize my OpenStack Fedora guest as its root file system merely had 5.4 GB to start with. So, I wanted to add atleast 15 GB more. After a bit of trial & error, here’s how I got it working. I’m using Fedora-19 in this case, but any other distro which supports libguestfs should be just fine.

Firstly, let’s check disk space inside the guest:

$ df -hT
    Filesystem              Type      Size  Used Avail Use% Mounted on
    /dev/mapper/fedora-root ext4      5.4G  5.4G     0 100% /
    devtmpfs                devtmpfs  4.7G     0  4.7G   0% /dev
    tmpfs                   tmpfs     4.7G     0  4.7G   0% /dev/shm
    tmpfs                   tmpfs     4.7G  392K  4.7G   1% /run
    tmpfs                   tmpfs     4.7G     0  4.7G   0% /sys/fs/cgroup
    tmpfs                   tmpfs     4.7G  472K  4.7G   1% /tmp
    /dev/vda1               ext4      477M   87M  365M  20% /boot
    /dev/loop0              ext4      4.6G   10M  4.4G   1% /srv/node/device1
    /dev/loop1              ext4      4.6G   10M  4.4G   1% /srv/node/device2
    /dev/loop2              ext4      4.6G   10M  4.4G   1% /srv/node/device3
    /dev/loop3              ext4      4.6G   10M  4.4G   1% /srv/node/device4
    /dev/vdb                ext4       17G   44M   16G   1% /mnt/newdisk

Print the libvirt XML to get the source of the disk

$ virsh dumpxml f19-test | grep -i source
      <source file='/var/lib/libvirt/images/f19-test.qcow2'/>

Above, I’m using a qcow2 disk image, I converted it to raw:

$ qemu-img convert -f qcow2 -O raw \
  /var/lib/libvirt/images/f19-test.qcow2 \
  /var/lib/libvirt/images/f19-test.raw

List the filesystems, partitions, block devices inside the raw disk image:

$ virt-filesystems --long --all -h -a \ 
  /var/lib/libvirt/images/f19-test.raw
Name              Type        VFS   Label  MBR  Size  Parent
/dev/sda1         filesystem  ext4  -      -    500M  -
/dev/fedora/root  filesystem  ext4  -      -    5.6G  -
/dev/fedora/swap  filesystem  swap  -      -    3.9G  -
/dev/fedora/root  lv          -     -      -    5.6G  /dev/fedora
/dev/fedora/swap  lv          -     -      -    3.9G  /dev/fedora
/dev/fedora       vg          -     -      -    9.5G  /dev/sda2
/dev/sda2         pv          -     -      -    9.5G  -
/dev/sda1         partition   -     -      83   500M  /dev/sda
/dev/sda2         partition   -     -      8e   9.5G  /dev/sda
/dev/sda          device      -     -      -    10G   -

Now, extend the file size of the raw disk image, using truncate:

# Create a new file based on original
$ truncate -r f19-test.raw f19-test.raw.new
# Adjust the new file size to 15G
$ truncate -s +15G f19-test.raw.new

List the file system partition info to find out the block device name:

$ virt-filesystems --partitions \
  --long -h -a f19-test.raw
Name       Type       MBR  Size  Parent
/dev/sda1  partition  83   500M  /dev/sda
/dev/sda2  partition  8e   9.5G  /dev/sda

Now, resize the new disk image using virt-resize. Note that, the --lv-expand option expands the root file system (thx Rich!):

$ virt-resize --expand /dev/sda2 --lv-expand \
  /dev/fedora/root f19-test.raw f19-test.raw.new
    Examining f19-test.raw ...
    **********
    
    Summary of changes:
    
    /dev/sda1: This partition will be left alone.
    
    /dev/sda2: This partition will be resized from 9.5G to 24.5G.  The LVM 
        PV on /dev/sda2 will be expanded using the 'pvresize' method.
    
    /dev/fedora/root: This logical volume will be expanded to maximum size. 
         The filesystem ext4 on /dev/fedora/root will be expanded using the 
        'resize2fs' method.
    
    **********
    Setting up initial partition table on f19-test.raw.new ...
    Copying /dev/sda1 ...
     100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00
    Copying /dev/sda2 ...
     100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00
    Expanding /dev/sda2 using the 'pvresize' method ...
    Expanding /dev/fedora/root using the 'resize2fs' method ...
    
    Resize operation completed with no errors.  Before deleting the old 
    disk, carefully check that the resized disk boots and works correctly.

Now, the size of the both new guests

$ ls -lash f19-test.raw f19-test.raw.new
2.7G -rw-r--r--. 1 qemu qemu 10G Apr 10 11:12 f19-test.raw
11G -rw-r--r--. 1 root root 25G Apr 10 12:13 f19-test.raw.new

Replace the old one w/ new one (you might want to take backup of the old one here, just in case):

$ mv f19-test.raw.new f19-test.raw

Also, update the libvirt XML file of the guest to reflect the raw disk image:

# Update source file path
$ virsh edit f19-test 
# grep the xml file to ensure.
$ grep source /etc/libvirt/qemu/f19-test.xml 
	
	

List file systems inside the newly created guest:

$ virt-filesystems --partitions --long -h -a f19-test.raw
Name       Type       MBR  Size  Parent
/dev/sda1  partition  83   500M  /dev/sda
/dev/sda2  partition  8e   25G   /dev/sda

Start the guest & ensure if everything looks sane:

$ virsh start f19-test --console

A few common operations while testing OpenStack on systemd based distros

While testing OpenStack on Fedora, I had to do some common operations to restart, list services. I started using systemctl — which is used to control & introspect the Systemd services. Noting them here, for reference. These should ideally be scripted, typing such long commands would be cumbersome.

List all enabled systemd unit files for OpenStack services:

$ systemctl list-unit-files --type=service | \
  grep -i openstack | grep -i enabled \
  | awk '{print $1}'

Status of OpenStack services active, inactive:

$ for i in `systemctl list-unit-files --type=service | \
  grep -i openstack |  grep -i enabled | \
  awk '{print $1}'`; do systemctl status $i ; done

Status of all OpenStack services, active:

$ for i in `systemctl list-unit-files --type=service | \
  grep -i openstack | grep -i enabled | awk '{print $1}'`; \
  do systemctl status $i ; done | grep Active -B1

Status of OpenStack services, inactive:

$ for i in `systemctl list-unit-files --type=service | \
  grep -i openstack | grep -i enabled | awk '{print $1}'`; \
  do systemctl status $i ; done | egrep -i 'Active|dead' -B1

Restart existing services. In systemd’s parlance, all enabled OpenStack unit files:

$ for i in `systemctl list-unit-files --type=service | \
  grep -i openstack | grep -i enabled | awk '{print $1}'`; \
  do systemctl restart $i ; done

Update: Updated the title to reflect its applicable for systemd based distros

Finding serial console log of a nova instance

The other day I was debugging why a volume booted nova instance doesn’t get dhcp leases. For that I need to find networking information of the instance during boot. Fortunately, nova attempts to log the boot details to a serial console log. To find where it’s located..

First, let’s list the instances which we care about:

$ nova list | grep f17
| 220d6e57-f7ad-4256-a861-7855ffc6788f | f17-builder       | ACTIVE  | novanetwork=192.168.32.4 |
| f1169841-5e67-45d9-85c4-0d44bd3bec90 | f17_volume_backed | ACTIVE | novanetwork=192.168.32.6 |

Then, grep that instance UUID in /etc/libvirt/qemu/* to find what’s the libvirt id of a specific nova instance:

$ grep f1169841-5e67-45d9-85c4-0d44bd3bec90 \
  /etc/libvirt/qemu/* 
/etc/libvirt/qemu/instance-00000007.xml:  f1169841-5e67-45d9-85c4-0d44bd3bec90
/etc/libvirt/qemu/instance-00000007.xml:      f1169841-5e67-45d9-85c4-0d44bd3bec90

$ virsh list | grep instance-00000007
 10    instance-00000007              running

From above, the libvirt instance is called instance-00000007

To find path to the serial console log of the instance, grep its libvirt xml information for the relevant fragment:

$ virsh dumpxml  instance-00000007 | grep -i console
      <source path='/var/lib/nova/instances/instance-00000007/console.log'/>
    <console type='file'>
      <source path='/var/lib/nova/instances/instance-00000007/console.log'/>
    </console>

The console log of the cirros image runs a bunch of useful debug commands after booting an instance — which gave me an insight that it wasn’t getting dhcp leases. Listing the commands it ran from the console.log for reference:

$ /etc/rc.d/init.d/sshd start
$ ifconfig -a
$ route -n
$ cat /etc/resolv.conf
# pinging nameservers
$ uname -a
$ dmesg | tail
$ tail -n 25 /var/log/messages

Also, for reference, relevant xml snippet of the console information:

$ virsh dumpxml instance-00000003 | \
  egrep -i 'serial type' -A9
    <serial type='file'>
      <source path='/var/lib/nova/instances/instance-00000003/console.log'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target port='1'/>
      <alias name='serial1'/>
    </serial>
    <console type='file'>
      <source path='/var/lib/nova/instances/instance-00000003/console.log'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>