Little more disk I/O perf. improvement with ‘fallocate’ing a qcow2 disk

[04-NOV-2015: Important update: Since this change in upstream QEMU, which introduces two new options preallocation=falloc and preallocation=full to qemu-img create, this is strongly recommended to use qemu-img create -f qcow2 preallocation=falloc […] to get the said performance benefits.]

Recently I’ve started using ‘preallocation=metadata’ flag while creating qcow2 disk images to extract some decent I/O performance. Today, while discussing qcow2 disk image performance with Stefan Hajnoczi (thank you!) on irc, I found, using fallocate — which preallocates all the blocks to a file — on a qcow2 disk image would improve disk I/O performance a little more as alls the blocks are allocated to the file ahead of time. (Just to note – fallocate comes w/ the linux standard pkg ‘util-linux-ng’)

Let’s run a quick test to see the disk I/O performance improvement by preallocating all the space in a qcow2 disk.

Create the disk image with ‘preallocation=metadata’

$ qemu-img create -f qcow2 -o preallocation=metadata /export/vmimgs/f16-test1.qcow2 8G
Formatting '/export/vmimgs/f16-test1.qcow2', fmt=qcow2 size=8589934592 encryption=off cluster_size=65536 preallocation='metadata' 

Let’s check the size of the image in bytes

$ ls -l /export/vmimgs/f16-test1.qcow2
-rw-r--r--. 1 root root 8591507456 Dec  2 16:55 /export/vmimgs/f16-test1.qcow2

# Also, print the allocated file size in blocks
$ ls -lash /export/vmimgs/f16-test1.qcow2
1.4M -rw-r--r--. 1 root root 8.1G Dec  2 16:55 /export/vmimgs/f16-test1.qcow2

Run fallocate to preallocate space to the disk image:

$ fallocate -l 8591507456 /export/vmimgs/f16-test1.qcow2 

Now, re-run ‘ls’ to print the allocated file size in blocks. (Notice that all the disk size, 8G, is now allocated.)

$ ls -lash /export/vmimgs/f16-test1.qcow2
8.1G -rw-r--r--. 1 root root 8.1G Dec  2 16:55 /export/vmimgs/f16-test1.qcow2

Also, let’s run ‘qemu-img info’ to get the disk size, virtual size.

$ qemu-img info f16-test1.qcow2 
image: f16-test1.qcow2
file format: qcow2
virtual size: 8.0G (8589934592 bytes)
disk size: 8.0G
cluster_size: 65536

As a simple test, I used the above disk image to create an @core only Fedora-16 guest(on a Fedora-16 host) and clocked the timing — it took roughly 5 min 32 sec to finish. While, previously, w/o fallocateing a disk image, when I clocked the same f-16 timing, it took nearly 8 minutes. So, there is a decent improvement noticed here.

With this, Stefan noted, disk write speed inside the guest machine should also be improved, when blocks are written for the first time. And also, due to less disk fragmentation — as all the space was preallocated in one operation — there would be fewer disk seeks during large read operations.



Filed under Uncategorized

7 responses to “Little more disk I/O perf. improvement with ‘fallocate’ing a qcow2 disk

  1. Nathan

    Great read, thank you! I am curious what your thoughts are on enabling io=native (Linux AIO) vs io=threads? I’m familiar with what the two are but not entirely knowledgeable on the differences, advantages and limitations of each. I’m running a few CentOS VM’s whose images live on an NFS mount. Currently following your directions in this post as well as setting cache to none and bus to Virtio. I appreciate any insight. Thank you again for everything!

    • Sorry, I’ve not experimented much w/ storage options of Linux AIO/Threads. But, I can point you to a discussioon of storage stack performance in KVM/QEMU at Linux Plumber’s 2010 which shows some metrics for aio=native vs aio=threads. Just google for – ‘KVM / QEMU Storage Stack Performance Discussion'(A pdf should come up). This is by Stefan Hajnoczi and Khoa Huynh. If you have spare systems, maybe you can experiment with both for a period of time, and see it goes :)

      Sorry for this delayed response(not much helpful though). I was away on a personal emergency for a while.

      • Yaniv

        – On NFS, one should use aio=threads. On block storage, aio=native.

        – You should add comparison of qcow2 fallocate’d against a raw-sparse image.

  2. Yaniv

    Also, for installation only, try with 1MB blocks.

  3. Nir

    Thank you for that Interesting post kashyapc.

  4. instead of preallocation=metadata use preallocation=full and you can skip using fallocate

    • Yep, that’s (‘full’ option) a recent improvement. There’s also the ‘preallocation=fallocate’ mode that was recently merged in upstream QEMU, that’ll use the posix_fallocate().

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s