Nested Virtualization — KVM, Intel, with VMCS Shadowing

[Previous installments on Nested Virtualization with KVM and Intel.]

This is part of some recent testing that I’ve been doing with upstream KVM (for 3.10.1). The threads linked here has initial tests bench-marking kernel compile (with make defconfig, a default config file) times in L2. And some minimal guestfish appliance start-up timings in L1.

Some details:

• Setup information to test with VMCS (Virtual Machine Control Structure) Shadowing. In brief, VMCS Shadowing — a processor specific feature — as described upstream, can reduce the overhead of nested virtualization by reducing the number of VMExits from L1 to L0.
• Simple scripts used to create L1 and L2.
• Libvirt XMLs of L1, L2 guests, for reference.

The gritty details of reasons for VMExits are described in Intel architecture manuals, Volume 3b, APPENDIX 1.

Fetching Rawhide Kernel builds using koji

To fetch rawhide kernels from Koji (Fedora build system), this is what I do:

Install the group “RPM Development Tools” to get koji packages

$ yum groupinstall "RPM Development Tools" 

NOTE: If you’re using versions < Fedora 19, please do: yum groupinstall "Development Tools"

Find the latest kernel in Rawhide

$ koji latest-pkg rawhide kernel                                                                      
Build                                     Tag                   Built by
----------------------------------------  --------------------  ----------------
kernel-3.10.0-0.rc0.git26.1.fc20          f20                   jwboyer

Download the resultant kernel from the o/p of the above command.

$ koji download-build \
  --arch=x86_64 kernel-3.10.0-0.rc0.git23.1.fc20 

Finally, install it

$ yum localinstall *.rpm 

Search for a specific patch from an upstream PULL request (KVM) in Fedora Rawhide Kernel

Before I use the latest rawhide kernel for a test, I had to ensure some specific patches made into it from KVM upstream GIT PULL request . This is how I tried:

Fetch the Rawhide Kernel SRPM

Install the rawhide SRPM:

 
# Move into SRPMS directory
$ cd ~/rpmbuild/SRPMS 

# Fetch the SRPM
$ wget http://kojipkgs.fedoraproject.org//packages/kernel/3.10.0/0.rc0.git23.1.fc20/src/kernel-3.10.0-0.rc0.git23.1.fc20.src.rpm 

# Install the kernel SRPM
$ rpm -ivh kernel-3.10.0-0.rc0.git23.1.fc20.src.rpm 

First, simpler way

git-describe to the rescue: Each Fedora rawhide kernel changelog has the precise output of Linus’ tree (thanks jwb!). So, let’s run it on the upstream kernel tree.

 
# Clone Linus' tree (or traverse to its path if it already exists)
# git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
$ cd linux 

# Run:
$ git describe
v3.9-11789-ge0fd9af

The above output can be understood as (from git-describe man page): The current head of the “parent” branch is based on v3.9, and is 11789 commits ahead of it; with a hash suffix ge0fd9af: where “-g” (stands for “git”), e0fd9af is the 7-character abbreviation for the tip commit (which was — e0fd9affeb64088eff407dfc98bbd3a5c17ea479)

Now, compare the above tag from the tag mentioned in changelog of Fedora Rawhide Kernel

$ rpm -q kernel --changelog | head -2 
* Thu May 09 2013 Josh Boyer  - 3.10.0-0.rc0.git23.1
- Linux v3.9-11789-ge0fd9af

Second, a more convoluted way to ensure a specific patch is in

Check upstream KVM tree, for the latest commit –
db6ae6158186a17165ef990bda2895ae7594b039

# Clone the upstream KVM git
$ git clone git://git.kernel.org/pub/scm/virt/kvm/kvm.git

# Traverse into its directory
$ cd kvm 

# Get the latest commit ID
$ git log tags/kvm-3.10-1 | head -1
db6ae6158186a17165ef990bda2895ae7594b039 

Now, get details the above commit. Its commit’s short description says: “kvm: Add compat_ioctl for device control API”

$  git log -p db6ae6158186a17165ef990bda2895ae7594b039 
.....
+       .compat_ioctl = kvm_device_ioctl, 

(that’s the first line of change)

Let’s grep for that string in our just installed kernel sources tree

$ cd ~/rpmbuild/SOURCES/ 

# Copy the patch file archive into a temp directory
$  cp patch-3.9-git23.xz /var/tmp/test/ 

# Extract & list it
$ unxz patch-3.9-git23.xz 
$ ls
patch-3.9-git23 

# grep for that specific line
$ grep ".compat_ioctl = kvm_device_ioctl" patch-3.9-git23 
+       .compat_ioctl = kvm_device_ioctl,

So, the above confirms that the patch (and the KVM GIT PULL request is contained in the koji rawhide build.

And the entire PULL request for KVM updates is merged with this commit ID – 01227a889ed56ae53aeebb9f93be9d54dd8b2de8.

Very late post — KVM Forum 2012 (incoherent) notes

While talking to Amit (thanks!) on IRC, he asked if I ever managed to take notes from the talks I attended at KVM Forum 2012 (side note – that page is up2date with slides from most talks). I did scribble some incoherent notes, but forgot to blog about it. Here we go — http://kashyapc.fedorapeople.org/virt/kvm-forum-2012-notes.txt

Resize a fedora 19 guest with libguestfs tools

The inimitable Rich Jones writes some incredibly useful software. I can’t count how many times they helped me debugging disk images, or gave great insights into different aspects of linux virtualization. One of those instances again — I had to resize my OpenStack Fedora guest as its root file system merely had 5.4 GB to start with. So, I wanted to add atleast 15 GB more. After a bit of trial & error, here’s how I got it working. I’m using Fedora-19 in this case, but any other distro which supports libguestfs should be just fine.

Firstly, let’s check disk space inside the guest:

$ df -hT
    Filesystem              Type      Size  Used Avail Use% Mounted on
    /dev/mapper/fedora-root ext4      5.4G  5.4G     0 100% /
    devtmpfs                devtmpfs  4.7G     0  4.7G   0% /dev
    tmpfs                   tmpfs     4.7G     0  4.7G   0% /dev/shm
    tmpfs                   tmpfs     4.7G  392K  4.7G   1% /run
    tmpfs                   tmpfs     4.7G     0  4.7G   0% /sys/fs/cgroup
    tmpfs                   tmpfs     4.7G  472K  4.7G   1% /tmp
    /dev/vda1               ext4      477M   87M  365M  20% /boot
    /dev/loop0              ext4      4.6G   10M  4.4G   1% /srv/node/device1
    /dev/loop1              ext4      4.6G   10M  4.4G   1% /srv/node/device2
    /dev/loop2              ext4      4.6G   10M  4.4G   1% /srv/node/device3
    /dev/loop3              ext4      4.6G   10M  4.4G   1% /srv/node/device4
    /dev/vdb                ext4       17G   44M   16G   1% /mnt/newdisk

Print the libvirt XML to get the source of the disk

$ virsh dumpxml f19-test | grep -i source
      <source file='/var/lib/libvirt/images/f19-test.qcow2'/>

Above, I’m using a qcow2 disk image, I converted it to raw:

$ qemu-img convert -f qcow2 -O raw \
  /var/lib/libvirt/images/f19-test.qcow2 \
  /var/lib/libvirt/images/f19-test.raw

List the filesystems, partitions, block devices inside the raw disk image:

$ virt-filesystems --long --all -h -a \ 
  /var/lib/libvirt/images/f19-test.raw
Name              Type        VFS   Label  MBR  Size  Parent
/dev/sda1         filesystem  ext4  -      -    500M  -
/dev/fedora/root  filesystem  ext4  -      -    5.6G  -
/dev/fedora/swap  filesystem  swap  -      -    3.9G  -
/dev/fedora/root  lv          -     -      -    5.6G  /dev/fedora
/dev/fedora/swap  lv          -     -      -    3.9G  /dev/fedora
/dev/fedora       vg          -     -      -    9.5G  /dev/sda2
/dev/sda2         pv          -     -      -    9.5G  -
/dev/sda1         partition   -     -      83   500M  /dev/sda
/dev/sda2         partition   -     -      8e   9.5G  /dev/sda
/dev/sda          device      -     -      -    10G   -

Now, extend the file size of the raw disk image, using truncate:

# Create a new file based on original
$ truncate -r f19-test.raw f19-test.raw.new
# Adjust the new file size to 15G
$ truncate -s +15G f19-test.raw.new

List the file system partition info to find out the block device name:

$ virt-filesystems --partitions \
  --long -h -a f19-test.raw
Name       Type       MBR  Size  Parent
/dev/sda1  partition  83   500M  /dev/sda
/dev/sda2  partition  8e   9.5G  /dev/sda

Now, resize the new disk image using virt-resize. Note that, the --lv-expand option expands the root file system (thx Rich!):

$ virt-resize --expand /dev/sda2 --lv-expand \
  /dev/fedora/root f19-test.raw f19-test.raw.new
    Examining f19-test.raw ...
    **********
    
    Summary of changes:
    
    /dev/sda1: This partition will be left alone.
    
    /dev/sda2: This partition will be resized from 9.5G to 24.5G.  The LVM 
        PV on /dev/sda2 will be expanded using the 'pvresize' method.
    
    /dev/fedora/root: This logical volume will be expanded to maximum size. 
         The filesystem ext4 on /dev/fedora/root will be expanded using the 
        'resize2fs' method.
    
    **********
    Setting up initial partition table on f19-test.raw.new ...
    Copying /dev/sda1 ...
     100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00
    Copying /dev/sda2 ...
     100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00
    Expanding /dev/sda2 using the 'pvresize' method ...
    Expanding /dev/fedora/root using the 'resize2fs' method ...
    
    Resize operation completed with no errors.  Before deleting the old 
    disk, carefully check that the resized disk boots and works correctly.

Now, the size of the both new guests

$ ls -lash f19-test.raw f19-test.raw.new
2.7G -rw-r--r--. 1 qemu qemu 10G Apr 10 11:12 f19-test.raw
11G -rw-r--r--. 1 root root 25G Apr 10 12:13 f19-test.raw.new

Replace the old one w/ new one (you might want to take backup of the old one here, just in case):

$ mv f19-test.raw.new f19-test.raw

Also, update the libvirt XML file of the guest to reflect the raw disk image:

# Update source file path
$ virsh edit f19-test 
# grep the xml file to ensure.
$ grep source /etc/libvirt/qemu/f19-test.xml 
	
	

List file systems inside the newly created guest:

$ virt-filesystems --partitions --long -h -a f19-test.raw
Name       Type       MBR  Size  Parent
/dev/sda1  partition  83   500M  /dev/sda
/dev/sda2  partition  8e   25G   /dev/sda

Start the guest & ensure if everything looks sane:

$ virsh start f19-test --console

A few common operations while testing OpenStack on systemd based distros

While testing OpenStack on Fedora, I had to do some common operations to restart, list services. I started using systemctl — which is used to control & introspect the Systemd services. Noting them here, for reference. These should ideally be scripted, typing such long commands would be cumbersome.

List all enabled systemd unit files for OpenStack services:

$ systemctl list-unit-files --type=service | \
  grep -i openstack | grep -i enabled \
  | awk '{print $1}'

Status of OpenStack services active, inactive:

$ for i in `systemctl list-unit-files --type=service | \
  grep -i openstack |  grep -i enabled | \
  awk '{print $1}'`; do systemctl status $i ; done

Status of all OpenStack services, active:

$ for i in `systemctl list-unit-files --type=service | \
  grep -i openstack | grep -i enabled | awk '{print $1}'`; \
  do systemctl status $i ; done | grep Active -B1

Status of OpenStack services, inactive:

$ for i in `systemctl list-unit-files --type=service | \
  grep -i openstack | grep -i enabled | awk '{print $1}'`; \
  do systemctl status $i ; done | egrep -i 'Active|dead' -B1

Restart existing services. In systemd’s parlance, all enabled OpenStack unit files:

$ for i in `systemctl list-unit-files --type=service | \
  grep -i openstack | grep -i enabled | awk '{print $1}'`; \
  do systemctl restart $i ; done

Update: Updated the title to reflect its applicable for systemd based distros

Finding serial console log of a nova instance

The other day I was debugging why a volume booted nova instance doesn’t get dhcp leases. For that I need to find networking information of the instance during boot. Fortunately, nova attempts to log the boot details to a serial console log. To find where it’s located..

First, let’s list the instances which we care about:

$ nova list | grep f17
| 220d6e57-f7ad-4256-a861-7855ffc6788f | f17-builder       | ACTIVE  | novanetwork=192.168.32.4 |
| f1169841-5e67-45d9-85c4-0d44bd3bec90 | f17_volume_backed | ACTIVE | novanetwork=192.168.32.6 |

Then, grep that instance UUID in /etc/libvirt/qemu/* to find what’s the libvirt id of a specific nova instance:

$ grep f1169841-5e67-45d9-85c4-0d44bd3bec90 \
  /etc/libvirt/qemu/* 
/etc/libvirt/qemu/instance-00000007.xml:  f1169841-5e67-45d9-85c4-0d44bd3bec90
/etc/libvirt/qemu/instance-00000007.xml:      f1169841-5e67-45d9-85c4-0d44bd3bec90

$ virsh list | grep instance-00000007
 10    instance-00000007              running

From above, the libvirt instance is called instance-00000007

To find path to the serial console log of the instance, grep its libvirt xml information for the relevant fragment:

$ virsh dumpxml  instance-00000007 | grep -i console
      <source path='/var/lib/nova/instances/instance-00000007/console.log'/>
    <console type='file'>
      <source path='/var/lib/nova/instances/instance-00000007/console.log'/>
    </console>

The console log of the cirros image runs a bunch of useful debug commands after booting an instance — which gave me an insight that it wasn’t getting dhcp leases. Listing the commands it ran from the console.log for reference:

$ /etc/rc.d/init.d/sshd start
$ ifconfig -a
$ route -n
$ cat /etc/resolv.conf
# pinging nameservers
$ uname -a
$ dmesg | tail
$ tail -n 25 /var/log/messages

Also, for reference, relevant xml snippet of the console information:

$ virsh dumpxml instance-00000003 | \
  egrep -i 'serial type' -A9
    <serial type='file'>
      <source path='/var/lib/nova/instances/instance-00000003/console.log'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target port='1'/>
      <alias name='serial1'/>
    </serial>
    <console type='file'>
      <source path='/var/lib/nova/instances/instance-00000003/console.log'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>

Multiple ways to access QEMU Monitor Protocol (QMP)

Once QEMU is built, to get a finer understanding of it, or even for plain old debugging, having familiarity with QMP (QEMU Monitor Protocol) is quite useful. QMP allows applications — like libvirt — to communicate with a running QEMU’s instance. There are a few different ways to access the QEMU monitor to query the guest, get device (eg: PCI, block, etc) information, modify the guest state (useful to understand the block layer operations) using QMP commands. This post discusses a few aspects of it.

Access QMP via libvirt’s qemu-monitor-command
Libvirt had this capability for a long time, and this is the simplest. It can be invoked by virsh — on a running guest, in this case, called ‘devstack’:

$ virsh qemu-monitor-command devstack \
--pretty '{"execute":"query-kvm"}'
{
    "return": {
        "enabled": true,
        "present": true
    },
    "id": "libvirt-8"
}

In the above example, I ran the simple command query-kvm which checks if (1) the host is capable of running KVM (2) and if KVM is enabled. Refer below for a list of possible ‘qeury’ commands.

QMP via telnet
To access monitor via any other way, we need to have qemu instance running in control mode, via telnet:

$ ./x86_64-softmmu/qemu-system-x86_64 \
  --enable-kvm -smp 2 -m 1024 \
  /export/images/el6box1.qcow2 \
  -qmp tcp:localhost:4444,server --monitor stdio
QEMU waiting for connection on: tcp::127.0.0.14444,server
VNC server running on `127.0.0.1:5900'
QEMU 1.4.50 monitor - type 'help' for more information
(qemu)

And, from a different shell, connect to that listening port 4444 via telnet:

$ telnet localhost 4444

Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
{"QMP": {"version": {"qemu": {"micro": 50, "minor": 4, "major": 1}, "package": ""}, "capabilities": []}}

We have to first enable QMP capabilities. This needs to be run before invoking any other commands, do:

{ "execute": "qmp_capabilities" }

QMP via unix socket
First, invoke the qemu binary in control mode using qmp, and create a unix socket as below:

$ ./x86_64-softmmu/qemu-system-x86_64 \
  --enable-kvm -smp 2 -m 1024 \
  /export/images/el6box1.qcow2 \
  -qmp unix:./qmp-sock,server --monitor stdio
QEMU waiting for connection on: unix:./qmp-sock,server

A few different ways to connect to the above qemu instance running in control mode, vi QMP:

1. Firstly, via nc :

   $ nc -U ./qmp-sock
   {"QMP": {"version": {"qemu": {"micro": 50, "minor": 4, "major": 1}, "package": ""}, "capabilities": []}}
   

But, with the above, you have to manually enable the QMP capabilities, and type each command in JSON syntax. It’s a bit cumbersome, & no history of commands typed is saved.

2. Next, a more simpler way — a python script called qmp-shell is located in the qemu source tree, under qemu/QMP, which hides some details — like manually running the qmp_capabilities.

   Connect to the unix socket using the qmp-shell script:

   $ ./qmp-shell ../qmp-sock 
   Welcome to the QMP low-level shell!
   Connected to QEMU 1.4.50
   
   (QEMU) 
   

   Then, just hit the key, and all the possible commands would be listed. To see a list of query commands:

   (QEMU) query-<TAB>
   query-balloon               query-commands              query-kvm                   query-migrate-capabilities  query-uuid
   query-block                 query-cpu-definitions       query-machines              query-name                  query-version
   query-block-jobs            query-cpus                  query-mice                  query-pci                   query-vnc
   query-blockstats            query-events                query-migrate               query-status                
   query-chardev               query-fdsets                query-migrate-cache-size    query-target                
   (QEMU) 
   

3. Finally, we can also acess the unix socket using socat and rlwrap. Thanks to upstream qemu developer Markus Armbruster for this hint.

   Invoke it this way, also execute a couple of commands — qmp_capabilities, and query-kvm, to view the response from the server.

   $ rlwrap -H ~/.qmp_history \
     socat UNIX-CONNECT:./qmp-sock STDIO
   {"QMP": {"version": {"qemu": {"micro": 50, "minor": 4, "major": 1}, "package": ""}, "capabilities": []}}
   {"execute":"qmp_capabilities"}
   {"return": {}}
   { "execute": "query-kvm" }
   {"return": {"enabled": true, "present": true}}
   

   Where, qmp_history contains recently ran QMP commands in JSON syntax. And rlwrap adds decent editing capabilities, recursive search & history. So, once you run all your commands, the ~/.qmp_history has a neat stack of all the QMP commands in JSON syntax.

   For instance, this is what my ~/.qmp_history file contains as I write this:

   $ cat ~/.qmp_history
   { "execute": "qmp_capabilities" }
   { "execute": "query-version" }
   { "execute": "query-events" }
   { "execute": "query-chardev" }
   { "execute": "query-block" }
   { "execute": "query-blockstats" }
   { "execute": "query-cpus" }
   { "execute": "query-pci" }
   { "execute": "query-kvm" }
   { "execute": "query-mice" }
   { "execute": "query-vnc" }
   { "execute": "query-spice " }
   { "execute": "query-uuid" }
   { "execute": "query-migrate" }
   { "execute": "query-migrate-capabilities" }
   { "execute": "query-balloon" }
   

To illustrate, I ran a few query commands (noted above) which provides an informative response from the server — no change is done to the state of the guest — so these can be executed safely.

I personally prefer the libvirt way, & accessing via unix socket with socat & rlwrap.

NOTE: To try each of the above variants, fisrst quit — type quit on the (qemu) shell — the qemu instance running in control mode, reinvoke it, then access it via one of the 3 different ways.

Debugging nova — a small illustration with pdb

The other day, I was testing non-ascii characters with nova commands. I started with adding an ssh key to nova with a non-ascii name, not realizing key-pair names won’t support them. This is how we ended up figuring it after a small debugging session with Flavio.

First, let’s try to add an ssh keypair & attempt to provide a non-ascii name:

$ nova keypair-add --pub_key ~/.ssh/id_rsa.pub äçë
ERROR: Keypair data is invalid (HTTP 400) (Request-ID: req-5a1a2683-d658-417b-a805-91346d5202dc)

Note – supplying --debug provides more info on the stdout. However, we’re interested to find out the root cause.

Let’s find out why the key-pair data is invalid by stepping through code using pdb. To start, stop the nova-api service, and ensure none of its processes are running :

$ service openstack-nova-api stop
$ ps aux | grep -i nova-api

Ensure to have the config attribute osapi_compute_workers set to 1 — I had it set to 5 while testing rhbz-81349 — so they won’t run in parallel, in /etc/nova/nova.conf :

$ grep osapi_compute_workers \
  /etc/nova/nova.conf | grep -v ^$ | grep -v ^# \
  osapi_compute_workers=1

Let’s set the pdb trace — import pdb; pdb.set_trace() — in the source code to break into the debugger — /usr/lib/python2.6/site-packages/nova/compute/api.py

Code context — I’m setting the breakpoint just after the self-explanatory function _validate_keypair_name :

.
.
class KeypairAPI(base.Base):
    """Sub-set of the Compute Manager API for managing key pairs."""
    def __init__(self, **kwargs):
        super(KeypairAPI, self).__init__(**kwargs)

    def _validate_keypair_name(self, context, user_id, key_name):

        import pdb; pdb.set_trace()       

        safechars = "_- " + string.digits + string.ascii_letters
        clean_value = "".join(x for x in key_name if x in safechars)
.
.

Now, invoke the nova-api service manually:

$ /usr/bin/nova-api --config-file \
  /usr/share/nova/nova-dist.conf --config-file \
  /etc/nova/nova.conf --logfile /var/log/nova/api.log

From a different terminal, again, try to create a keypair with non-ascii characters — this time, it’ll let us break into the debugger

$ nova keypair-add --pub-key ~/.ssh/id_rsa.pub äçë

Now, from shell from where we invoked the nova-api server daemon, a debugger Pdb would be waiting, to step through the code, do:

• type n — which continues execution until the next line in the current function is reached
• print the value safechars — indicates what values are considered safe — which is what we’re trying to find out
• quit the debugger

.
.
.
2013-03-26 19:56:26 71243 DEBUG nova.utils [-] Reloading cached file /etc/nova/policy.json read_cached_file /usr/lib/python2.6/site-packages/nova/uti1
2013-03-26 19:56:27 INFO nova.api.openstack.wsgi [req-9ef1fd31-0cec-4465-894a-f1ae0fa9a77e 320ce46de7e24a75a7ff8906d7355ff7 57ff99aae24b4035b52177a72s
> /usr/lib/python2.6/site-packages/nova/compute/api.py(2198)_validate_keypair_name()
-> safechars = "_- " + string.digits + string.ascii_letters
(Pdb) 
(Pdb)  n
> /usr/lib/python2.6/site-packages/nova/compute/api.py(2199)_validate_keypair_name()
-> clean_value = "".join(x for x in key_name if x in safechars)
(Pdb)  safechars
 '_- 0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ' 
(Pdb) 
(Pdb)  quit

So, the above indicates non-ascii characters are not allowed for key-pair names.

Clean up

Remove the debugging fragment from the source file. Start the nova-api server daemon gracefully –

 $ service openstack-nova-api start 

OpenStack — nova image-create, under the hood

I was trying to understand what kind of image nova image-create creates. It’s not entirely obvious from its help output, which says — Creates a new image by taking a snapshot of a running server. But what kind of snapshot? let’s figure.

nova image-create operations

The command is invoked as below

 $  nova image-create fed18 "snap1-of-fed18" --poll  

Drilling into nova’s source code — nova/virt/libvirt/driver.py — this is what image-create does:

1. If the guest — based on which snapshot is to be taken — is running, nova calls libvirt’s virsh managedsave, which saves and stops a running guest, to be restarted later from the saved state.
2. Next, it creates a qcow2 internal disk snapshot of the guest (now offline).
3. Then, extracts the internal named snapshot from the qcow2 file & exports it to a RAW format and temporarily places in $instances_path/snapshots.
4. Deletes the internal named snapshot from the qcow2 file.
5. Finally, uploads that image into OpenStack glance service — which can be confirmed by running glance image-list.

A simple test
To get a bit more clarity, let’s try nova’s actions on a single qocw2 disk — with a running Fedora 18 OS — using libvirt’s shell virsh and QEMU’s qemu-img:

 
# Save the state and stop a running guest
$ virsh managedsave fed18 

# Create a qemu internal snapshot
$ qemu-img snapshot -c snap1 fed18.qcow2 

# Get information about the disk
$ qemu-img info fed18.qcow2 

# Extract the internal snapshot, 
# convert it to raw and export it a file
$ qemu-img convert -f qcow2 -O raw -s \
    snap1 fed18.qcow2 snap1-fed18.img 

# Get information about the new image
# extracted from the snapshot
$ qemu-img info snap1-fed18.img 

# List out file sizes of the original 
# and the snapshot
$ ls -lash fed18.qcow2 snap1-fed18.qcow2 

# Delete the internal snapshot 
# from the original disk
$ qemu-img snapshot -d snap1 fed18.qcow2 

# Again, get information of the original disk
$ qemu-img info fed18.qcow2 

# Start the guest again
$ virsh start fed18 

Thanks to Nikola Dipanov for helping me on where to look.

Update: A few things I missed to mention (thanks again for comments from Nikola) — I was using libvirt, kvm as underlying hypervisor technologies, with OpenStack Folsom release.