Little more disk I/O perf. improvement with ‘fallocate’ing a qcow2 disk

Recently I’ve started using ‘preallocation=metadata’ flag while creating qcow2 disk images to extract some decent I/O performance. Today, while discussing qcow2 disk image performance with Stefan Hajnoczi (thank you!) on irc, I found, using fallocate — which preallocates all the blocks to a file — on a qcow2 disk image would improve disk I/O performance a little more as alls the blocks are allocated to the file ahead of time. (Just to note – fallocate comes w/ the linux standard pkg ‘util-linux-ng’)

Let’s run a quick test to see the disk I/O performance improvement by preallocating all the space in a qcow2 disk.

Create the disk image with ‘preallocation=metadata’

 
$ qemu-img create -f qcow2 -o preallocation=metadata /export/vmimgs/f16-test1.qcow2 8G
Formatting '/export/vmimgs/f16-test1.qcow2', fmt=qcow2 size=8589934592 encryption=off cluster_size=65536 preallocation='metadata' 
 

Let’s check the size of the image in bytes

$ ls -l /export/vmimgs/f16-test1.qcow2
-rw-r--r--. 1 root root 8591507456 Dec  2 16:55 /export/vmimgs/f16-test1.qcow2

# Also, print the allocated file size in blocks
$ ls -lash /export/vmimgs/f16-test1.qcow2
1.4M -rw-r--r--. 1 root root 8.1G Dec  2 16:55 /export/vmimgs/f16-test1.qcow2
 

Run fallocate to preallocate space to the disk image:

$ fallocate -l 8591507456 /export/vmimgs/f16-test1.qcow2 
 

Now, re-run ‘ls’ to print the allocated file size in blocks. (Notice that all the disk size, 8G, is now allocated.)

$ ls -lash /export/vmimgs/f16-test1.qcow2
8.1G -rw-r--r--. 1 root root 8.1G Dec  2 16:55 /export/vmimgs/f16-test1.qcow2
$ 
 

Also, let’s run ‘qemu-img info’ to get the disk size, virtual size.

$ qemu-img info f16-test1.qcow2 
image: f16-test1.qcow2
file format: qcow2
virtual size: 8.0G (8589934592 bytes)
disk size: 8.0G
cluster_size: 65536
$ 
 

As a simple test, I used the above disk image to create an @core only Fedora-16 guest(on a Fedora-16 host) and clocked the timing — it took roughly 5 min 32 sec to finish. While, previously, w/o fallocateing a disk image, when I clocked the same f-16 timing, it took nearly 8 minutes. So, there is a decent improvement noticed here.

With this, Stefan noted, disk write speed inside the guest machine should also be improved, when blocks are written for the first time. And also, due to less disk fragmentation — as all the space was preallocated in one operation — there would be fewer disk seeks during large read operations.

Snapshotting with libvirt for qcow2 images

Libvirt 0.9.6 was recently out. Take a look at 0.9.5 changelog for truckload of features/bugfixes/cleanups(specifically snapshot related) from the libvirt team.

So, I grabbed the F14 srpm from Libvirt ftp, and made a quick Fedora koji scratch build of libvirt-0.9.6 for Fedora 15 and gave the snapshot features a whirl. Here it goes:

(Also noted below is some very useful discussion I had(on #virt, OFTC) with Eric Blake (Upstream/Red Hat’s Libvirt dev, very friendly guy.) on snapshots. It was way informative not to capture it.)

Context on types of snapshots
At the moment, snapshotting in KVM/QEMU/Libvirt land is supported primarily for QCOW2 disk images. I briefly discussed about Qcow2 previously here.

There are several different types of snapshots possible. Some idea on that:

Internal snapshot: A type of snapshot, where a single QCOW2 file will hold both the ‘saved state’ and the ‘delta’ since that saved point. ‘Internal snapshots’ are very handy because it’s only a single file where all the snapshot info. is captured, and easy to copy/move around the machines.

External snapshot: Here, the ‘original qcow2 file’ will be in a ‘read-only’ saved state, and the new qcow2 file(which will be generated once snapshot is created) will be the *delta* for the changes. So, all the changes will now be written to this delta file. ‘External Snapshots’ are useful for performing backups. Also, external snapshot creates a qcow2 file with the original file as its backing image, and the backing file can be /read/ in parallel with the running qemu.

VM State: This will save the guest/domain state to a file. So, if you take a snapshot including VM state, we can then shut off that guest and use the freed up memory for other purposes on the host or for other guests. Internally this calls qemu monitor’s ‘savevm’ command. Note that this only takes care of VM state(and not disk snapshot). To try this out:

 
# Memory before saving the guest f15vm3
##############################################
[root@moon ~]# free -m
             total       used       free     shared    buffers     cached
Mem:         10024       5722       4301          0        164       4445
-/+ buffers/cache:       1112       8911
Swap:            0          0          0
##############################################
[root@moon ~]# virsh list
 Id Name                 State
----------------------------------
  5 f15guest             running
  6 f15vm3               running
##############################################
# Save the guest f15vm3 to a file 'foof15vm3'
[root@moon ~]# virsh save f15vm3 foof15vm3
Domain f15vm3 saved to foof15vm3
##############################################
# Now, f15vm3 is gracefully saved/shutdown.
[root@moon ~]# virsh list
 Id Name                 State
----------------------------------
  5 f15guest             running
##############################################
# Notice the RAM being freed
[root@moon ~]# free -m
             total       used       free     shared    buffers     cached
Mem:         10024       5418       4605          0        164       4493
-/+ buffers/cache:        760       9263
Swap:            0          0          0
##############################################
# Let's restore the guest back from the file 'foof15vm3'
[root@moon ~]# virsh restore foof15vm3 
Domain restored from foof15vm3
##############################################
# list the status. f15vm3 is up and running.
[root@moon ~]# virsh list
 Id Name                 State
----------------------------------
  5 f15guest             running
  7 f15vm3               running
##############################################

For brevity, let’s try out internal disk snapshots where all the snapshot info. (like disk and VM state info) are stored in a single qcow2 file.
Virsh(libvirt shell interface to manage guests) has some neat options for snapshot supports. So, I’ve got an F15 guest (Qcow2 disk image).

Internal Disk Snapshots when the guest is online/running

For illustration purpose, let’s use a Fedora-15 guest called ‘f15guest’ .

[root@moon ~]# virsh list 
 Id Name                 State
----------------------------------
  4 f15guest             running

[root@moon ~]# 

For clarity, ensure there are no prior snapshot instances around.

[root@moon ~]# virsh snapshot-list f15guest
 Name                 Creation Time             State
------------------------------------------------------------

[root@moon ~]# 
 

Before creating a snapshot, we need to create a snapshot xml file with 2 simple elements (name and description) if you need sensible name for the snapshot. Note that only these two fields are user settable. Rest of the info. will be filled by Libvirt.

[root@moon ~]#  cat /var/tmp/snap1-f15guest.xml
<domainsnapshot>
    <name>snap1-f15pki </name>
    <description>F15 system with dogtag pki packages </description>
</domainsnapshot>

[root@moon ~]# 

Eric Blake noted that, the domainsnapshot xml file is optional now for ‘snapshot-create’ if you don’t need a description for the snapshot. And if it’s okay with libvirt generating the snapshot name for us. (More on this, refer below)

Now, I’m taking a snapshot while the ‘guest’ is running live. Here, Eric noted that, especially when running/live, the more RAM the guest has, and the more active the guest is modifying that RAM, the longer the it will take to create a snapshot. This was a guest was mostly an idle guest.

[root@moon ~]# virsh snapshot-create f15guest /var/tmp/snap1-f15guest.xml 
Domain snapshot snap1-f15pki  created from '/var/tmp/snap1-f15guest.xml'
[root@moon ~]# 

While the snapshot-creation is in _progress_ on the live guest, the state of the guest will be ‘paused’.

[root@moon ~]# virsh list
 Id Name                 State
----------------------------------
  4 f15guest             paused

[root@moon ~]# 

Once, the snapshot is created, list the snapshots of f15guest

[root@moon ~]# virsh snapshot-list f15guest
 Name                 Creation Time             State
------------------------------------------------------------
 snap1-f15pki         2011-10-04 19:04:00 +0530 running

[root@moon ~]#

Internal snapshot while the guest is offline

For fun, I created another snapshot, but _after_ shutting down the guest. Now, the snapshot creation is just instantaneous.

[root@moon ~]# virsh list
 Id Name                 State
----------------------------------

[root@moon ~]# 
[root@moon ~]# virsh snapshot-create f15guest
Domain snapshot 1317757628 created
[root@moon ~]# 

List the snapshots of ‘f15guest’ using virsh.

[root@moon ~]# virsh snapshot-list f15guest
 Name                 Creation Time             State
------------------------------------------------------------
 1317757628           2011-10-05 01:17:08 +0530 shutoff
 snap1-f15pki         2011-10-04 19:04:00 +0530 running

To see some information about the VM size, snapshot info:

[root@moon ~]# qemu-img info /export/vmimgs/f15guest.qcow2 
image: /export/vmimgs/f15guest.qcow2
file format: qcow2
virtual size: 8.0G (8589934592 bytes)
disk size: 3.2G
cluster_size: 65536
Snapshot list:
ID        TAG                 VM SIZE                DATE       VM CLOCK
1         snap1-f15pki           1.7G 2011-10-04 19:04:00   32:06:34.974
2         1317757628                0 2011-10-05 01:17:08   00:00:00.000
[root@moon ~]# 

To revert to a particular snapshot, virsh snapshot-revert domain snapshotname

Also, discussed with Eric, in what cases does virsh invoke Qemu’s ‘savevm‘ and ‘qemu-img snapshot -c‘ commands while creating different types of snapshots discussed earlier above. Here is the outline:

- it uses ‘qemu-img snapshot -c‘ if the domain is offline and –disk-only was not specified
- it uses qemu’s ‘savevm‘ if the domain is online and –disk-only was not specified
- it uses qemu’s ‘snapshot_blkdev‘ if the domain is online and –disk-only is specified

(Note: –disk-only is an option to capture only ‘disk state’ but not VM state. This option is available w/ virsh ‘snapshot-create’ or ‘snapshot-create-as’ commands.)

Thanks Eric for the detail.

Creating a Qcow2 virtual machine

Qcow2 disk image is an interesting format which supports features like internal and external snapshots, backing files, image compression, encryption. But also, it’s I/O performance is very slow compared to RAW format. Here are a couple of settings which can extract reasonable performance out of Qcow2 disk images.

Create a qcow2 disk image
First, let’s create a qcow2 disk image using ‘qemu-img’ tool

$ /usr/bin/qemu-img create -f qcow2 -o preallocation=metadata /export/vmimgs/glacier.qcow2 8G

NOTE: At this point in time, preallocation=metadata option is the best we can do to extract max. possible (near RAW) I/O performance out of QCOW2 format. (hint from Kevin Wolf – Qemu/Qcow2 developer )

From the below listing that 970M is the allocated or used size of the guest, 8.1G is the max size the image can ‘grow to’.

[root@moon tmp]# ls -lash /export/vmimgs/glacier.qcow2 
970M -rw-r--r--. 1 qemu qemu 8.1G Sep 24 23:45 /export/vmimgs/glacier.qcow2
[root@moon tmp]# 

Create the guest

# Create an unattended minimal guest install using a qcow2 disk image
virt-install --connect=qemu:///system \
    --network=bridge:br0 \
    --initrd-inject=/var/tmp/fed-minimal.ks \
    --extra-args="ks=file:/fed-minimal.ks console=tty0 console=ttyS0,115200" \
    --name=glacier \
    --disk path=/export/vmimgs/glacier.qcow2,format=qcow2 \
    --ram 2048 \
    --vcpus=2 \
    --check-cpu \
    --hvm \
    --location=http://download.fedora.redhat.com/pub/fedora/linux/releases/15/Fedora/x86_64/os/ \
    --nographics 

The above will create a minimal guest w/ a qcow2 disk image format. Content of the fed-minimal kickstart is here

Once, the guest is created, ensure to have cache=’none’ parameter in ‘disk’ element of the guest’s xml file (if not present, add it and redefine the xml. It looks like below). This is another aspect which can improve the disk I/O performance.

[root@moon ~]# grep cache /etc/libvirt/qemu/glacier.xml -A 4 -B 1
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/export/vmimgs/glacier.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
[root@moon ~]# virsh define /etc/libvirt/qemu/glacier.xml
Domain glacier defined from /etc/libvirt/qemu/glacier.xml
[root@moon ~]# virsh start glacier
Domain glacier started

[root@moon ~]#

I’m still trying to wrap my head around the caching and preallocation mechanisms of the qcow2 format. Meanwhile, work on Qcow2 version-3 is in progress in upstream qemu.