Multiple ways to access QEMU Monitor Protocol (QMP)

Once QEMU is built, to get a finer understanding of it, or even for plain old debugging, having familiarity with QMP (QEMU Monitor Protocol) is quite useful. QMP allows applications — like libvirt — to communicate with a running QEMU’s instance. There are a few different ways to access the QEMU monitor to query the guest, get device (eg: PCI, block, etc) information, modify the guest state (useful to understand the block layer operations) using QMP commands. This post discusses a few aspects of it.

Access QMP via libvirt’s qemu-monitor-command
Libvirt had this capability for a long time, and this is the simplest. It can be invoked by virsh — on a running guest, in this case, called ‘devstack’:

$ virsh qemu-monitor-command devstack \
--pretty '{"execute":"query-kvm"}'
{
    "return": {
        "enabled": true,
        "present": true
    },
    "id": "libvirt-8"
}

In the above example, I ran the simple command query-kvm which checks if (1) the host is capable of running KVM (2) and if KVM is enabled. Refer below for a list of possible ‘qeury’ commands.

QMP via telnet
To access monitor via any other way, we need to have qemu instance running in control mode, via telnet:

$ ./x86_64-softmmu/qemu-system-x86_64 \
  --enable-kvm -smp 2 -m 1024 \
  /export/images/el6box1.qcow2 \
  -qmp tcp:localhost:4444,server --monitor stdio
QEMU waiting for connection on: tcp::127.0.0.14444,server
VNC server running on `127.0.0.1:5900'
QEMU 1.4.50 monitor - type 'help' for more information
(qemu)

And, from a different shell, connect to that listening port 4444 via telnet:

$ telnet localhost 4444

Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
{"QMP": {"version": {"qemu": {"micro": 50, "minor": 4, "major": 1}, "package": ""}, "capabilities": []}}

We have to first enable QMP capabilities. This needs to be run before invoking any other commands, do:

{ "execute": "qmp_capabilities" }

QMP via unix socket
First, invoke the qemu binary in control mode using qmp, and create a unix socket as below:

$ ./x86_64-softmmu/qemu-system-x86_64 \
  --enable-kvm -smp 2 -m 1024 \
  /export/images/el6box1.qcow2 \
  -qmp unix:./qmp-sock,server --monitor stdio
QEMU waiting for connection on: unix:./qmp-sock,server

A few different ways to connect to the above qemu instance running in control mode, vi QMP:

  1. Firstly, via nc :

    $ nc -U ./qmp-sock
    {"QMP": {"version": {"qemu": {"micro": 50, "minor": 4, "major": 1}, "package": ""}, "capabilities": []}}
    
  2. But, with the above, you have to manually enable the QMP capabilities, and type each command in JSON syntax. It’s a bit cumbersome, & no history of commands typed is saved.

  3. Next, a more simpler way — a python script called qmp-shell is located in the qemu source tree, under qemu/QMP, which hides some details — like manually running the qmp_capabilities.

    Connect to the unix socket using the qmp-shell script:

    $ ./qmp-shell ../qmp-sock 
    Welcome to the QMP low-level shell!
    Connected to QEMU 1.4.50
    
    (QEMU) 
    

    Then, just hit the key, and all the possible commands would be listed. To see a list of query commands:

    (QEMU) query-<TAB>
    query-balloon               query-commands              query-kvm                   query-migrate-capabilities  query-uuid
    query-block                 query-cpu-definitions       query-machines              query-name                  query-version
    query-block-jobs            query-cpus                  query-mice                  query-pci                   query-vnc
    query-blockstats            query-events                query-migrate               query-status                
    query-chardev               query-fdsets                query-migrate-cache-size    query-target                
    (QEMU) 
    
  4. Finally, we can also acess the unix socket using socat and rlwrap. Thanks to upstream qemu developer Markus Armbruster for this hint.

    Invoke it this way, also execute a couple of commands — qmp_capabilities, and query-kvm, to view the response from the server.

    $ rlwrap -H ~/.qmp_history \
      socat UNIX-CONNECT:./qmp-sock STDIO
    {"QMP": {"version": {"qemu": {"micro": 50, "minor": 4, "major": 1}, "package": ""}, "capabilities": []}}
    {"execute":"qmp_capabilities"}
    {"return": {}}
    { "execute": "query-kvm" }
    {"return": {"enabled": true, "present": true}}
    

    Where, qmp_history contains recently ran QMP commands in JSON syntax. And rlwrap adds decent editing capabilities, recursive search & history. So, once you run all your commands, the ~/.qmp_history has a neat stack of all the QMP commands in JSON syntax.

    For instance, this is what my ~/.qmp_history file contains as I write this:

    $ cat ~/.qmp_history
    { "execute": "qmp_capabilities" }
    { "execute": "query-version" }
    { "execute": "query-events" }
    { "execute": "query-chardev" }
    { "execute": "query-block" }
    { "execute": "query-blockstats" }
    { "execute": "query-cpus" }
    { "execute": "query-pci" }
    { "execute": "query-kvm" }
    { "execute": "query-mice" }
    { "execute": "query-vnc" }
    { "execute": "query-spice " }
    { "execute": "query-uuid" }
    { "execute": "query-migrate" }
    { "execute": "query-migrate-capabilities" }
    { "execute": "query-balloon" }
    

To illustrate, I ran a few query commands (noted above) which provides an informative response from the server — no change is done to the state of the guest — so these can be executed safely.

I personally prefer the libvirt way, & accessing via unix socket with socat & rlwrap.

NOTE: To try each of the above variants, fisrst quit — type quit on the (qemu) shell — the qemu instance running in control mode, reinvoke it, then access it via one of the 3 different ways.

Leave a Comment

Filed under Uncategorized

Debugging nova — a small illustration with pdb

The other day, I was testing non-ascii characters with nova commands. I started with adding an ssh key to nova with a non-ascii name, not realizing key-pair names won’t support them. This is how we ended up figuring it after a small debugging session with Flavio.

First, let’s try to add an ssh keypair & attempt to provide a non-ascii name:

$ nova keypair-add --pub_key ~/.ssh/id_rsa.pub äçë
ERROR: Keypair data is invalid (HTTP 400) (Request-ID: req-5a1a2683-d658-417b-a805-91346d5202dc)

Note – supplying --debug provides more info on the stdout. However, we’re interested to find out the root cause.

Let’s find out why the key-pair data is invalid by stepping through code using pdb. To start, stop the nova-api service, and ensure none of its processes are running :

$ service openstack-nova-api stop
$ ps aux | grep -i nova-api

Ensure to have the config attribute osapi_compute_workers set to 1 — I had it set to 5 while testing rhbz-81349 — so they won’t run in parallel, in /etc/nova/nova.conf :

$ grep osapi_compute_workers \
  /etc/nova/nova.conf | grep -v ^$ | grep -v ^# \
  osapi_compute_workers=1

Let’s set the pdb trace — import pdb; pdb.set_trace() — in the source code to break into the debugger — /usr/lib/python2.6/site-packages/nova/compute/api.py

Code context — I’m setting the breakpoint just after the self-explanatory function _validate_keypair_name :

.
.
class KeypairAPI(base.Base):
    """Sub-set of the Compute Manager API for managing key pairs."""
    def __init__(self, **kwargs):
        super(KeypairAPI, self).__init__(**kwargs)

    def _validate_keypair_name(self, context, user_id, key_name):

        import pdb; pdb.set_trace()       

        safechars = "_- " + string.digits + string.ascii_letters
        clean_value = "".join(x for x in key_name if x in safechars)
.
.

Now, invoke the nova-api service manually:

$ /usr/bin/nova-api --config-file \
  /usr/share/nova/nova-dist.conf --config-file \
  /etc/nova/nova.conf --logfile /var/log/nova/api.log

From a different terminal, again, try to create a keypair with non-ascii characters — this time, it’ll let us break into the debugger

$ nova keypair-add --pub-key ~/.ssh/id_rsa.pub äçë

Now, from shell from where we invoked the nova-api server daemon, a debugger Pdb would be waiting, to step through the code, do:

  • type n — which continues execution until the next line in the current function is reached
  • print the value safechars — indicates what values are considered safe — which is what we’re trying to find out
  • quit the debugger
.
.
.
2013-03-26 19:56:26 71243 DEBUG nova.utils [-] Reloading cached file /etc/nova/policy.json read_cached_file /usr/lib/python2.6/site-packages/nova/uti1
2013-03-26 19:56:27 INFO nova.api.openstack.wsgi [req-9ef1fd31-0cec-4465-894a-f1ae0fa9a77e 320ce46de7e24a75a7ff8906d7355ff7 57ff99aae24b4035b52177a72s
> /usr/lib/python2.6/site-packages/nova/compute/api.py(2198)_validate_keypair_name()
-> safechars = "_- " + string.digits + string.ascii_letters
(Pdb) 
(Pdb)  n
> /usr/lib/python2.6/site-packages/nova/compute/api.py(2199)_validate_keypair_name()
-> clean_value = "".join(x for x in key_name if x in safechars)
(Pdb)  safechars
 '_- 0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ' 
(Pdb) 
(Pdb)  quit

So, the above indicates non-ascii characters are not allowed for key-pair names.

Clean up

Remove the debugging fragment from the source file. Start the nova-api server daemon gracefully –

 $ service openstack-nova-api start 

2 Comments

Filed under Uncategorized

OpenStack — nova image-create, under the hood

I was trying to understand what kind of image nova image-create creates. It’s not entirely obvious from its help output, which says — Creates a new image by taking a snapshot of a running server. But what kind of snapshot? let’s figure.

nova image-create operations

The command is invoked as below

 $  nova image-create fed18 "snap1-of-fed18" --poll  

Drilling into nova’s source code — nova/virt/libvirt/driver.py — this is what image-create does:

  1. If the guest — based on which snapshot is to be taken — is running, nova calls libvirt’s virsh managedsave, which saves and stops a running guest, to be restarted later from the saved state.
  2. Next, it creates a qcow2 internal disk snapshot of the guest (now offline).
  3. Then, extracts the internal named snapshot from the qcow2 file & exports it to a RAW format and temporarily places in $instances_path/snapshots.
  4. Deletes the internal named snapshot from the qcow2 file.
  5. Finally, uploads that image into OpenStack glance service — which can be confirmed by running glance image-list.

A simple test
To get a bit more clarity, let’s try nova’s actions on a single qocw2 disk — with a running Fedora 18 OS — using libvirt’s shell virsh and QEMU’s qemu-img:

 
# Save the state and stop a running guest
$ virsh managedsave fed18 

# Create a qemu internal snapshot
$ qemu-img snapshot -c snap1 fed18.qcow2 

# Get information about the disk
$ qemu-img info fed18.qcow2 

# Extract the internal snapshot, 
# convert it to raw and export it a file
$ qemu-img convert -f qcow2 -O raw -s \
    snap1 fed18.qcow2 snap1-fed18.img 

# Get information about the new image
# extracted from the snapshot
$ qemu-img info snap1-fed18.img 

# List out file sizes of the original 
# and the snapshot
$ ls -lash fed18.qcow2 snap1-fed18.qcow2 

# Delete the internal snapshot 
# from the original disk
$ qemu-img snapshot -d snap1 fed18.qcow2 

# Again, get information of the original disk
$ qemu-img info fed18.qcow2 

# Start the guest again
$ virsh start fed18 

Thanks to Nikola Dipanov for helping me on where to look.

Update: A few things I missed to mention (thanks again for comments from Nikola) — I was using libvirt, kvm as underlying hypervisor technologies, with OpenStack Folsom release.

2 Comments

Filed under Uncategorized

Building qemu from upstream git source

From time to time, I test from upstream qemu to try out some specific area that I’m interested in (lately, block layer operations). Here’s how I try to build qemu from upstream git source, for a specific target — x86_64.

Install development tools (I’m using a Fedora 18 machine here). And a couple of other development libraries

$  yum groupinstall "Development Tools" 
$  yum install glib2-devel zlibrary zlib-devel -y 

Clone the qemu upstream source tree

$  git clone git://git.qemu.org/qemu.git  
$  cd qemu 

Run the qemu configure script for x86_64 architecture, and run make

$ ./configure --target-list=x86_64-softmmu \
--disable-werror --enable-debug 
$  make -j5 

Note: When the qemu configure script is invoked, it’ll throw an error asking you to either install the pixman package or fetch its git submodule.

I fetched pixman submodule

 $  git submodule update --init pixman 

If you have an OS disk image handy, you can invoke the newly built qemu binary to boot a guest on the stdio, assuming there’s a serial console enabled — having console=tty0 console=ttyS0,115200 inside your guest’s kernel command line in /etc/grub2.conf

$ ./x86_64-softmmu/qemu-system-x86_64 --enable-kvm \
-smp 2 -m 1024 /export/images/fedora18.qcow2 \
-nographic

Note: With the above invocation, --enable-kvm parameter will make use of existing kvm kernel module from the system.

7 Comments

Filed under Uncategorized

OpenStack nova — dealing with unused base images

Nova uses disk image cache management to handle unused base disk images, validate their integrity, etc. In this entry, let’s try to find out how unused base images are cleaned up. For context on Openstack’s libvirt base images, take a look at Pádraig’s excellent write-up on life of an OpenStack libvirt image. And a prior related post on disk image usage by nova.

Set the relevant config directives
Let’s set periodic_interval=1 — the no. of seconds between running periodic tasks; image_cache_manager_interval=1 — the no. of periodic scheduler ticks to wait between runs of the image cache manager. I’ve set both of them to ’1′ for test purpose, and grepped to ensure the change reflects


#-----#
[tbox ~keystone_user1]$ sudo echo -e "periodic_interval=1\nimage_cache_manager_interval=1" >> /etc/nova/nova.conf
#-----#
[tbox ~keystone_user1]$ sudo egrep '^image_cache_manager_interval|^periodic_interval' /etc/nova/nova.conf 
periodic_interval=1
image_cache_manager_interval=1
[tbox ~keystone_user1]$ 
#-----#

Restart all openstack nova services, so the config changes take effect:


[root@tbox ~(keystone_user1)]# for j in `for i in $(ls -1 /etc/init.d/openstack-nova-*) ; do $i status | grep running ; done | awk '{print $1}'` ; do service $j restart ; done

Delete a running nova instance, verify logs to see the effect of config changes

Let’s stop and delete the only nova instance running — so that nova’s image cache manager knows the image is no longer in use, and the previously enabled config options can take effect.


#-----#
[tbox ~keystone_user1]$ nova list
+--------------------------------------+---------+--------+-------------------+
| ID                                   | Name    | Status | Networks          |
+--------------------------------------+---------+--------+-------------------+
| 11c1997a-b1ff-46ef-b96c-f329576de8d6 | fedora4 | ACTIVE | net1=10.xx.yyy.zz |
+--------------------------------------+---------+--------+-------------------+
[tbox ~keystone_user1]$ nova stop 11c1997a-b1ff-46ef-b96c-f329576de8d6
#-----#
[tbox ~keystone_user1]$ nova delete 11c1997a-b1ff-46ef-b96c-f329576de8d6
[tbox ~keystone_user1]$
#-----#
[tbox ~keystone_user1]$ nova list 
#-----#
[tbox ~keystone_user1]$ sudo virsh list --all
 Id    Name                           State
----------------------------------------------------

List out the base_images (now they should become invalid) :


[tbox ~keystone_user1]$ sudo ls -lash /var/lib/nova/instances/_base/
total 1.7G
4.0K drwxr-xr-x. 2 nova nova 4.0K Feb 18 00:41 .
4.0K drwxr-xr-x. 3 nova nova 4.0K Feb 18 04:01 ..
739M -rw-r--r--. 1 nova nova 9.8G Feb 18 04:01 06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3
741M -rw-r--r--. 1 nova nova  20G Feb 18 04:01 06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20
241M -rw-r--r--. 1 nova nova 241M Feb 18 00:41 06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3.part
[tbox ~keystone_user1]$ 

At this point, the — now invalid — base_images should’ve been deleted. Let’s look at nova’s compute logs, location: /var/log/nova/compute.log


.
.
2012-02-18 04:13:03 37747 INFO nova.compute.resource_tracker [-] Compute_service record updated for interceptor.foo.bar.com 
2012-02-18 04:13:03 37747 WARNING nova.virt.libvirt.imagecache [-] Unknown base file: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3
2012-02-18 04:13:03 37747 WARNING nova.virt.libvirt.imagecache [-] Unknown base file: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20
2012-02-18 04:13:03 37747 INFO nova.virt.libvirt.imagecache [-] Removable base files: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3 /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20
[tbox ~keystone_user1]$

Now, it says, base-files are too young to remove:


.
.
012-01-18 04:16:46 37747 INFO nova.virt.libvirt.imagecache [-] Base file too young to remove: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3
2012-02-18 04:16:46 37747 INFO nova.virt.libvirt.imagecache [-] Base file too young to remove: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20

Let’s drill down a bit further, by searching in upstream nova code for how young is ‘too young’:


kashyap@test$ git clone git://github.com/openstack/nova.git && cd nova
kashyap@nova$ grep FLAGS.remove_unused_original_minimum_age_seconds nova/virt/libvirt/imagecache.py
            maxage = FLAGS.remove_unused_original_minimum_age_seconds
kashyap@nova$
kashyap@nova$ git grep remove_unused_original_minimum_age_seconds
etc/nova/nova.conf.sample:# remove_unused_original_minimum_age_seconds=86400
nova/virt/libvirt/imagecache.py:    cfg.IntOpt('remove_unused_original_minimum_age_seconds',
nova/virt/libvirt/imagecache.py:            maxage = FLAGS.remove_unused_original_minimum_age_seconds
kashyap@nova$

Ok, 86400 seconds (24 hours) appears to be the default time. Let’s, set that value to 60 seconds in nova.conf and restart all nova services again:


#-----#
[tbox ~keystone_user1]$ sudo echo -e "remove_unused_original_minimum_age_seconds=60" >> /etc/nova/nova.conf
#-----#
[tbox ~keystone_user1]$ sudo egrep '^image_cache_manager_interval|^periodic_interval|^remove_unused_original_minimum_age_seconds' /etc/nova/nova.conf 
periodic_interval=1
image_cache_manager_interval=1
remove_unused_original_minimum_age_seconds=60
[tbox ~keystone_user1]$ 
#-----#

Restart all nova services again:


[tbox ~keystone_user1]$ sudo -i
[root@tbox ~(keystone_user1)]# for j in `for i in $(ls -1 /etc/init.d/openstack-nova-*) ; do $i status | grep running ; done | awk '{print $1}'` ; do service $j restart ; done

Now, the /var/log/compute/compute.log indicates one of the base-files being removed, that’s good:


.
.
2012-02-18 04:24:17 41389 WARNING nova.virt.libvirt.imagecache [-] Unknown base file: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810
a0d1d5d3_20
2012-02-18 04:24:17 41389 INFO nova.virt.libvirt.imagecache [-] Removable base files: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810
a0d1d5d3 /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20
2012-02-18 04:24:17 41389 INFO nova.virt.libvirt.imagecache [-] Removing base file: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3

Let’s check in the nova base images directory. Now, it seems like, two more base_images are still remaining:


[tbox ~keystone_user1]$ sudo ls -1 /var/lib/nova/instances/_base/
06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20
06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3.part
[tbox ~keystone_user1]$ 

Let’s get back to nova compute.log, it still says ‘Unknown base file’ and ‘Base file too young to remove’:


.
.
2012-02-18 04:50:45 41389 WARNING nova.virt.libvirt.imagecache [-] Unknown base file: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20
2012-02-18 04:50:45 41389 INFO nova.virt.libvirt.imagecache [-] Removable base files: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20
2012-02-18 04:50:45 41389 INFO nova.virt.libvirt.imagecache [-] Base file too young to remove: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20

Let’s check the nova compute.log again — now it’s indicating one of the other two unused base images being deleted — from time-stamps, in 10 minutes.


.
.
2012-02-18 05:01:24 41389 INFO nova.compute.resource_tracker [-] Compute_service record updated for interceptor.foo.bar.com 
2012-02-18 05:01:24 41389 WARNING nova.virt.libvirt.imagecache [-] Unknown base file: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20
2012-02-18 05:01:24 41389 INFO nova.virt.libvirt.imagecache [-] Removable base files: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20
2012-02-18 05:01:24 41389 INFO nova.virt.libvirt.imagecache [-] Removing base file: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20

Now, the only remaining one appears to be a .part image.


[tbox ~keystone_user1]$ sudo ls -1 /var/lib/nova/instances/_base/
06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3.part
[tbox ~keystone_user1]$

From further investigation by Padraig , the stale .part files turned out to be a different bug (for which he posted a fix upstream) — When converting non-raw (in this case, qcow2) to raw, libvirt leaves behind the original non-raw image on disk.

Finally
Let’s turn the config directives back to sane defaults, and comment out the config directives periodic_interval, image_cache_manager_interval. And, restart all nova services to take the config change effect.


#-----#
[tbox ~keystone_user1]$  sudo sed -i 's/remove_unused_original_minimum_age_seconds=60/remove_unused_originalminimum_age_seconds=86400/' foobar1.txt
#-----#
[tbox ~keystone_user1]$  sudo sed -i 's/periodic_interval=1/\#periodic_interval=1/;s/image_cache_manager_interval=1/\#image_cache_manager_interval=1/' >> /etc/nova/nova.conf
#-----#
[root@tbox ~(keystone_user1)]# for j in `for i in $(ls -1 /etc/init.d/openstack-nova-*) ; do $i status | grep running ; done | awk '{print $1}'` ; do service $j restart ; done
#-----#

UPDATE: I also had to set the config directive remove_unused_resized_minimum_age_seconds=60 in nova.conf, to remove the re-sized — according to the flavor chosen, in this case, it’s m1.small — raw images from the _base directory.

1 Comment

Filed under Uncategorized

A couple of caveats while setting up devstack on Fedora-18

It took more than a couple of attempts to get a running devstack instance on Fedora-18. I was following danpb’s notes. Here are a few tweaks I had to do get it working:

That’s the localrc file I had, before proceeding ahead:


$ cat localrc 
DESTDIR=$HOME/src/openstack
DATA_DIR=$DESTDIR/data

LOGFILE=$DATA_DIR/logs/stack.log
SCREEN_LOGDIR=$DATA_DIR/logs

# Switch to use QPid instead of RabbitMQ 
disable_service rabbit
enable_service qpid

# Replace with your primary interface name
HOST_IP_IFACE=eth0
PUBLIC_INTERFACE=eth0
VLAN_INTERFACE=eth0
FLAT_INTERFACE=eth0

# Replace with whatever password you wish to use
MYSQL_PASSWORD=testpwd
SERVICE_TOKEN=testpwd
SERVICE_PASSWORD=testpwd
ADMIN_PASSWORD=testpwd

# Pre-populate glance with a minimal image and a Fedora 17 image
IMAGE_URLS="http://launchpad.net/cirros/trunk/0.3.0/+download/cirros-0.3.0-x86_64-uec.tar.gz,http://berrange.fedorapeople.org/images/2012-11-15/f17-x86_64-openstack-sda.qcow2"
$ 

After running stack.sh, you might see some failures — like tgtd service failing to start. The below is what I had in my tgtd.conf and stack.conf.


====
# rpm -q scsi-target-utils
scsi-target-utils-1.0.32-2.fc18.x86_64
====
$ cat /etc/tgt/tgtd.conf 
# The default config file
include /etc/tgt/targets.conf

# Config files from other packages etc.
#include /etc/tgt/conf.d/*.conf

# Explicitly import the stack.conf file
include /etc/tgt/conf.d/stack.conf
====
$ cat /etc/tgt/conf.d/stack.conf 
include /home/tuser1/src/openstack/data/cinder/volumes/*
$
====

And restart tgtd service:


$ systemctl restart tgtd.service  
$ systemctl status tgtd.service                                                                                                                                             
tgtd.service - tgtd iSCSI target daemon
          Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled)
          Active: active (running) since Wed 2013-02-27 06:28:13 EST; 5s ago
         Process: 26826 ExecStop=/usr/sbin/tgtadm --op delete --mode system (code=exited, status=0/SUCCESS)
         Process: 26822 ExecStop=/usr/sbin/tgt-admin --update ALL -c /dev/null (code=exited, status=0/SUCCESS)
         Process: 26820 ExecStop=/usr/sbin/tgtadm --op update --mode sys --name State -v offline (code=exited, status=0/SUCCESS)
         Process: 26888 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v ready (code=exited, status=0/SUCCESS)
         Process: 26882 ExecStartPost=/usr/sbin/tgt-admin -e -c $TGTD_CONFIG (code=exited, status=0/SUCCESS)
         Process: 26880 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v offline (code=exited, status=0/SUCCESS)
        Main PID: 26879 (tgtd)
          CGroup: name=systemd:/system/tgtd.service
                  └─26879 /usr/sbin/tgtd -f

Feb 27 06:28:12 devstack-fedenglabfoobarcom tgtd[26879]: librdmacm: Warning: couldn't read ABI version.
Feb 27 06:28:12 devstack-fedenglabfoobarcom tgtd[26879]: librdmacm: Warning: assuming: 4
Feb 27 06:28:12 devstack-fedenglabfoobarcom tgtd[26879]: librdmacm: Fatal: unable to get RDMA device list
Feb 27 06:28:12 devstack-fedenglabfoobarcom tgtd[26879]: tgtd: iser_ib_init(3376) Failed to initialize RDMA; load kernel modules?
Feb 27 06:28:12 devstack-fedenglabfoobarcom tgtd[26879]: tgtd: work_timer_start(146) use timer_fd based scheduler
Feb 27 06:28:12 devstack-fedenglabfoobarcom tgtd[26879]: tgtd: bs_init(313) use signalfd notification
Feb 27 06:28:13 devstack-fedenglabfoobarcom systemd[1]: Started tgtd iSCSI target daemon.

Source the openrc file which will do few things: configure keystone authentication credentials, set up nova’s compute api version, etc:


$ . openrc 
$ 

Let’s list images in Glance:


$ glance image-list
+--------------------------------------+---------------------------------+-------------+------------------+-----------+--------+
| ID                                   | Name                            | Disk Format | Container Format | Size      | Status |
+--------------------------------------+---------------------------------+-------------+------------------+-----------+--------+
| fa1adc47-1ff3-4394-99b3-e782a27762b0 | cirros-0.3.0-x86_64-uec         | ami         | ami              | 25165824  | active |
| b6137e19-fe48-40a1-81d8-ccbbae6bfc2b | cirros-0.3.0-x86_64-uec-kernel  | aki         | aki              | 4731440   | active |
| 3780a3bd-e3ad-4323-908e-8f22789e44e1 | cirros-0.3.0-x86_64-uec-ramdisk | ari         | ari              | 2254249   | active |
| 0c445018-f23c-4e5a-876b-912ea8f6c636 | f17-x86_64-openstack-sda        | qcow2       | bare             | 251985920 | active |
+--------------------------------------+---------------------------------+-------------+------------------+-----------+--------+
$ 

Add a new key pair:


$ nova keypair-add oskey > oskey.priv
$ ls
accrc    eucarc      exercises    extras.d  functions    lib      localrc  oskey.priv  rejoin-stack.sh  stackrc         stack.sh  tools
AUTHORS  exerciserc  exercise.sh  files     HACKING.rst  LICENSE  openrc   README.md   samples          stack-screenrc  tests     unstack.sh
$ 

Boot a flavor:


$ nova boot --key-name oskey --image f17-x86_64-openstack-sda  --flavor m1.tiny f17demo1                                                                           
+-----------------------------+--------------------------------------+
| Property                    | Value                                |
+-----------------------------+--------------------------------------+
| status                      | BUILD                                |
| updated                     | 2013-02-27T11:58:58Z                 |
| OS-EXT-STS:task_state       | scheduling                           |
| key_name                    | oskey                                |
| image                       | f17-x86_64-openstack-sda             |
| hostId                      |                                      |
| OS-EXT-STS:vm_state         | building                             |
| flavor                      | m1.tiny                              |
| id                          | 5eaf23b5-3558-4b07-8195-9d24734beced |
| security_groups             | [{u'name': u'default'}]              |
| user_id                     | f28638e924a645e1b69650b8acc99283     |
| name                        | f17demo1                             |
| adminPass                   | e9LZqPFdBvjq                         |
| tenant_id                   | 9b54eaecedb149de9d1d48c43210d463     |
| created                     | 2013-02-27T11:58:58Z                 |
| OS-DCF:diskConfig           | MANUAL                               |
| metadata                    | {}                                   |
| accessIPv4                  |                                      |
| accessIPv6                  |                                      |
| progress                    | 0                                    |
| OS-EXT-STS:power_state      | 0                                    |
| OS-EXT-AZ:availability_zone | None                                 |
| config_drive                |                                      |
+-----------------------------+--------------------------------------+
$ 

Now, list the running nova instances:

 
$ nova list
+--------------------------------------+----------+--------+----------+
| ID                                   | Name     | Status | Networks |
+--------------------------------------+----------+--------+----------+
| 5eaf23b5-3558-4b07-8195-9d24734beced | f17demo1 | ERROR  |          |
+--------------------------------------+----------+--------+----------+
$ 

Oh, it says ERROR, let’s take a look at logs.

NOTE1: Log files to see: screen-n-cpu.log, screen-n-sch.log (located in $HOME/src/openstack/devstack/data/logs).
NOTE2: Use ‘less -r’ to view the above log files, so they’re presented in a readable format. Otherwise all control characters (ESC, Carriage return) are displayed in caret notataion, and it’s unpleasant to look at.

So, I missed to add executable bit to $HOME directory, let’s set it:


$ sudo chmod -R +x /home/tuser1/
$ 

Let’s delete the old instance, boot a flavor (in this case, a tiny one – 512 MB memory), & list the running nova instance, and ensure it’s ACTIVE:

 
#----------#
$ nova delete 5eaf23b5-3558-4b07-8195-9d24734beced                                                                                                                 
$ 
#----------#
$ nova boot --key-name oskey --image f17-x86_64-openstack-sda  --flavor m1.tiny f17demo1
#----------#
$ nova list
+--------------------------------------+----------+--------+------------------+
| ID                                   | Name     | Status | Networks         |
+--------------------------------------+----------+--------+------------------+
| 5710ba30-499e-4c2a-872c-327217e229c6 | f17demo1 | ACTIVE | private=10.0.0.3 |
+--------------------------------------+----------+--------+------------------+
$ 
#----------#

List the running openstack services:

 
$ openstack-status 
== Support services ==
mysqld:                       active (disabled on boot)
libvirtd:                     active
qpidd:                        active
$ 

Leave a Comment

Filed under Uncategorized

Nova’s way of using a disk image when it boots a guest for first time

Let’s see how Nova — the compute component of OpenStack — uses a disk image from the time it was initially imported into Glance (OpenStack’s repository for virtual machine disk images) till a new virtual machine instance is booted and running.

I started by downloading a Fedora-17 disk image from here
(Keep a note of the size of the image — 241 MB )


[tuser1@interceptor ~(keystone_user1)]$ ls -lash f17-x86_64-openstack-sda.qcow2 
241M -rw-rw-r--. 1 tuser1 tuser1 241M Jan 13  2012 f17-x86_64-openstack-sda.qcow2
[tuser1@interceptor ~(keystone_user1)]$ 

Import the above Fedora 17 disk image into Glance:


[tuser1@interceptor ~(keystone_admin)]$ glance image-create --name="fedora-17" --is-public=true \
--disk-format=qcow2 --container-format bare < f17-x86_64-openstack-sda.qcow2


(Note that I have to source Keystone admin credentials to list images from Glance).

Let’s list the current images in Glance:


tuser1@interceptor ~(keystone_admin)]$ glance image-list
+--------------------------------------+-----------+-------------+------------------+------------+--------+
| ID                                   | Name      | Disk Format | Container
Format | Size       | Status |
+--------------------------------------+-----------+-------------+------------------+------------+--------+
| 1e6292f9-82bd-4cdb-969e-c863cb1c6692 | fedora-17 | qcow2       | bare
| 251985920  | active |
| acc4c853-9153-4e80-b3c8-e253451ae983 | rhel63    | qcow2       | bare
| 1074135040 | active |
+--------------------------------------+-----------+-------------+------------------+------------+--------+
[tuser1@interceptor ~(keystone_admin)]$ 

List current running Nova instances :


[tuser1@interceptor ~(keystone_user1)]$ nova list
+--------------------------------------+-----------+--------+-------------------+
| ID                                   | Name      | Status | Networks          |
+--------------------------------------+-----------+--------+-------------------+
| 08d616a9-87a1-4c0d-b986-7d6aa5ed6780 | fedora-t1 | ACTIVE | net1=ww.xx.yyy.zz |
| 3e487977-37e8-4f26-9443-d65ecbdf83c9 | fedora-t2 | ACTIVE | net1=ww.xx.yyy.zz |
| 48d9e518-a91f-48db-9d9b-965b243e7113 | fedora-t4 | ACTIVE | net1=ww.xx.yyy.zz |
+--------------------------------------+-----------+--------+-------------------+
[tuser1@interceptor ~(keystone_user1)]$ 

Let’s also list the guests running using libvirt’s virsh :


[tuser1@interceptor ~(keystone_user1)]$ sudo virsh list
 Id    Name                           State
----------------------------------------------------
 12    instance-0000000c              running
 13    instance-0000000d              running
 22    instance-00000012              running

[tuser1@interceptor ~(keystone_user1)]$ 

Find the block device in use for one of the running instances to examine it further — instance-0000000c, in this case:


[tuser1@interceptor ~(keystone_user1)]$ sudo virsh domblklist instance-0000000c
Target     Source
------------------------------------------------
vda        /var/lib/nova/instances/instance-0000000c/disk

[tuser1@interceptor ~(keystone_user1)]$ 

Let’s get information about the disk in use by the above nova instance; specifically, find its backing file:


[tuser1@interceptor ~(keystone_user1)]$ qemu-img info /var/lib/nova/instances/instance-0000000c/disk
image: /var/lib/nova/instances/instance-0000000c/disk
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 149M
cluster_size: 65536
backing file: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20

Now, get information about the “backing file” used by the overlay (used by the running Nova instance above):


[tuser1@interceptor ~(keystone_user1)]$ qemu-img info /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20
image: /var/lib/nova/instances/_base/06a057b9c7b0b27e3b496f53d1e88810a0d1d5d3_20
file format: raw
virtual size: 20G (21474836480 bytes)
disk size: 740M
[tuser1@interceptor ~(keystone_user1)]$ 


It’s worth noting here that the original disk image initially uploaded into glance was a qcow2 image. However, the base image being used by Nova is a raw disk image.

From this, we can see: When booting a virtual machine instance for the first time, Nova does a few things:

  1. Make a copy of the original qcow2 disk image from Glance, convert it into a ‘raw’ sparse image & make it a base image (its location – /var/lib/nova/instances/_base) — Reason for turning non-raw to raw, refer “Background info” section below.
  2. Expands the size of the base image to 20GB (because, I used the m1.small flavour from Nova, when initially booting the image.)
  3. Use this base image & instantiate a copy on write overlay (qcow2) to boot the nova instance.

To demonstrate this post, I used Red Hat OpenStack Folsom, RHEL 6.4, single node, running these services — nova, glance, keystone, cinder; using KVM as the underlying hypervisor.

Background info:

[1] From nova’s git log, the conversion from non ‘raw’ images to ‘raw’ is tracked down to this commit

commit ff9d353b2f4fee469e530fbc8dc231a41f6fed84
Author: Scott Moser 
Date:   Mon Sep 19 16:57:44 2011 -0400

    convert images that are not 'raw' to 'raw' during caching to node
    
    This uses 'qemu-img' to convert images that are not 'raw' to be 'raw'.
    By doing so, it
     a.) refuses to run uploaded images that have a backing image reference
         (LP: #853330, CVE-2011-3147)
     b.) ensures that when FLAGS.use_cow_images is False, and the libvirt
         xml written specifies 'driver_type="raw"' that the disk referenced
         is also raw format. (LP: #837102)
     c.) removes compression that might be present to avoid cpu bottlenecks
         (LP: #837100)
    
    It does have the negative side affect of using more space in the case where
    the user uploaded a qcow2 (or other advanced image format) that could have
    been used directly by the hypervisor.  That could, later, be remedied by
    another 'qemu-img convert' being done to the "preferred" format of the
    hypervisor.

[2] Pádraig Brady pointed out (thank you !) this bz — https://bugs.launchpad.net/nova/+bug/932180 to note further reasons for converting images from non raw to raw.

As an aside, more informaion on disk image size allocation/ performance improvements upstream – https://blueprints.launchpad.net/nova/+spec/preallocated-images

UPDATE : Pádraig Brady writes in excellent detail about the life of an openstack libvirt image — describing Openstack — http://www.pixelbeat.org/docs/openstack_libvirt_images/

3 Comments

Filed under Uncategorized