When using QCOW2-based external snapshots, it is desirable to reduce an entire disk image chain to a single disk to retain performance and increase while the guest is running. Upstream QEMU and libvirt has recently acquired the ability to do that. Relevant git commits for QEMU (Jeff Cody) and libvirt (Eric Blake).
This is best illustrated with a quick example.
Let’s start with the below disk image chain as below for a guest called vm1. For simplicity’s sake:
[base] <-- [sn1] <-- [sn2] <-- [current] (live QEMU)
Once live active block commit operation is complete (step 5 below), the result will be a flattened disk image chain where data from sn1, sn2 and current are live commited into base:
[base] (live QEMU)
(1) List the current active image in use:
$ virsh domblklist vm1 Target Source ------------------------------------------------ vda /export/images/base.qcow2
(2) For a quick test, create external snapshots. (And, repeat the above operation two more times, so we have the chain: [base] <– [sn1] <– [sn2] <– [current] )
$ virsh snapshot-create-as \ --domain vm1 snap1 \ --diskspec vda,file=/export/images/sn1.qcow2 \ --disk-only --atomic
(3) Enumerate the backing file chain:
$ qemu-img info --backing-chain current.qcow2 [. . .] # output discarded for brevity
(4) Again, check the current active disk image:
$ virsh domblklist vm1 Target Source ------------------------------------------------ vda /export/images/current.qcow2
(5) Live Active commit an entire chain, including pivot:
$ virsh blockcommit vm1 vda \ --active --pivot --verbose Block Commit: [100 %] Successfully pivoted
Explanation:
- –active: It performs a two stage operation: first stage – it commits the contents from top images into base (i.e. sn1, sn2, current into base); in the second stage, the block operation remains awake to synchronize any further changes (from top images into base), here the user can take two actions: cancel the job, or pivot the job, i.e. adjust the base image as the current active image.
- –pivot: Once data is committed from sn1, sn2 and current into base, it pivots the live QEMU to use base as the active image.
- –verbose: Displays a progress of block operation.
- Finally, the disk image backing chain is shortened to a single disk image.
(6) Optionally, list the current active image in use. It’s now back to ‘base’ which has all the contents from current, sn2, sn1):
$ virsh domblklist vm1 Target Source ------------------------------------------------ vda /export/images/base.qcow2
Any idea when this will be made available in the current Ubuntu repos? If not, or if it is a ways away do you know if you lose any useful functionality by compiling from source vs. using the version in the repos? I know for example that when compiling qemu on CentOS you lose the rhel machine types, but I don’t really know performance wise if this makes any real difference. I really like your articles on virtualization by the way. There is one article in particular that I reference constantly. Good work.
About when it’s available in Ubuntu, I’m afraid I don’t follow it, so you’ll probably get a better response from the Ubuntu virt maintainers :-)
On compiling from source, I only test on Fedora, it’s a bit bleeding edge indeed. Here’s some notes on how I do it on Fedora[1]. I don’t see any immediate reasons for performance degradation. But, if you do hit any issues in your test environment, feel free to write to libvirt-users@redhat.com list.
Thanks for the comment!
[1] http://kashyapc.com/2014/03/10/notes-for-building-kvm-based-virtualization-components-from-upstream-git/
Pingback: KVM – Live backups with qcow2 | Gonzalo Marcote | Open source, open mind
Hey,
thanks for your post, that’s exactly what I was searching for. However I get the following error while executing the blockcommit:
error: internal error: unable to execute QEMU command ‘block-commit’: Could not reopen file: Permission denied
I can’t find anything helpful in the logs, dmesg shows only some apparmor profile messages:
audit: type=1400 audit(1422927949.728:88): apparmor=”STATUS” operation=”profile_replace” profile=”unconfined” name=”libvirt-ac89c746-90c4-ae6e-9604-6ddeb112062b” pid=18131 comm=”apparmor_parser”
audit: type=1400 audit(1422927949.748:89): apparmor=”STATUS” operation=”profile_replace” profile=”unconfined” name=”qemu_bridge_helper” pid=18131 comm=”apparmor_parser”
audit: type=1400 audit(1422927950.128:90): apparmor=”STATUS” operation=”profile_replace” profile=”unconfined” name=”libvirt-ac89c746-90c4-ae6e-9604-6ddeb112062b” pid=18133 comm=”apparmor_parser”
audit: type=1400 audit(1422927950.144:91): apparmor=”STATUS” operation=”profile_replace” profile=”unconfined” name=”qemu_bridge_helper” pid=18133 comm=”apparmor_parser”
qemu-system-x86_64 –version
QEMU emulator version 2.1.2 (Debian 1:2.1+dfsg-11ubuntu2), Copyright (c) 2003-2008 Fabrice Bellard
libvirtd –version
libvirtd (libvirt) 1.2.8
uname -a
Linux 3.18.0-9-generic #10-Ubuntu SMP Mon Jan 12 21:41:54 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Ubuntu 15.04 Server
Any idea what could be wrong?
You haven’t noted the exact CLI you used with blockcommit. Most likely QEMU process doesn’t have access the disk images in question. So, ensure permissions are set so that QEMU has access ($
chmod go+rx $Root-directory-that-contains-images
)Also check libvirtd logs: /var/log/libvirt/qemu/$VM.log.
Please note: For topics like these, it’s better discussed on libvirt-users mailing list (https://www.redhat.com/archives/libvirt-users/) so others can benefit from it too.
Hi Kashyap. Thanks for your posts. Combined with http://wiki.libvirt.org/, they help me quite a lot setting up my VM backups.
One detail still bugs me.
I want to create a disk snapshot, then copy the base file, then commit and delete the snapshot. AFAIU, this is pretty much the only use case for disk snapshots.
My VM is called prod. Here is what I’d do:
# virsh snapshot-create-as –domain prod snap –diskspec vda,file=/srv/vm/snap.qcow2 –disk-only –atomic
# cp /srv/vm/prod.qcow2 /path/to_backup/dir/backup_file
# virsh blockcommit prod vda –active –pivot
# rm /srv/vm/snap.qcow2
But then libvirt keeps a “trace” of snap (.xml file) and I can’t delete it with virsh delete. Actually, I can delete it by manually removing snap.xml file, but this does not sound satisfying.
As suggested on http://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit, I tried the ‘–no-metadata’ option:
# virsh snapshot-create-as –domain prod snap –diskspec vda,file=/srv/vm/snap.qcow2 –disk-only –atomic –no-metadata
and it seems to do the trick.
Is this the right way to go or am I missing something?
When I do so, snap is not listed by
#virsh snapshot-list prod
Is this an issue?
If this is the right way to proceed, what is the use case for the command without the ‘–no-metadata’ option, which leaves us with a snapshot we can’t get rid of?
Thanks.
Hi Jérôme,
Glad you find the posts useful.
Firstly, live disk backup without guest down time (using a combination of disk-only snapshot + live merging the overlay back into the base with
blockcommit
) is not the only use case for disk snapshots. There are several other use cases for external disk snapshots besides that: you can use a common base image & create multiple overlays, allowing you to quickly provision a cluster from it; live storage migration; “Optimized QMP performance” (quoting Eric Blake).About the “trace” of the snapshot file, as you guessed, yes, manually deleting is not the right way — despite removing the snap.xml file (from /var/lib/libvirt/qemu/snapshot),
virsh snapshot-list
will still enumerate ‘snap’, because you’ve removed the file behind libvirt’s back. As I noted in the wiki page, the--no-metadata
parameter is the correct way to do it; and,virsh snapshot-list
not listing the ‘snap’ is correct behaviour (and not an “issue”), because you explicitly told libvirt to not track it. So, the latter method is the right way to proceed.The use case for
snapshot-create-as
command without the--no-metadata
option is that it trivially allows you to _revert_ to a snapshot (in ‘internal snapshots’ case). Support for reverting to an external snapshot is in-progress upstream.Hi. Thank you for taking the time to answer.
> About the “trace” of the snapshot file, as you guessed, yes, manually deleting is not the right way
What would be the correct way to get rid of the snapshot, then, assuming it was not created with ‘–no-metadata’ option?
Is my setup in a dangerously wrong state since I’ve been removing .qcow2 and .xml snapshot files manually after the blockcommit?
> despite removing the snap.xml file (from /var/lib/libvirt/qemu/snapshot), virsh snapshot-list will still enumerate ‘snap’, because you’ve removed the file behind libvirt’s back.
I shall check, but I’m pretty sure the snapshot is not enumerated anymore by `virsh snapshot-list prod` once the .xml file is moved away and libvirtd restarted.
> The use case for snapshot-create-as command without the –no-metadata option is that it trivially allows you to _revert_ to a snapshot (in ‘internal snapshots’ case). Support for reverting to an external snapshot is in-progress upstream.
I was in doubt because the wiki page is about external disk snapshots and it reads “Optionally, you can also supply ‘–no-metadata’ option”, suggesting the “normal” way to do external snapshots does not necessary involves that option.
Anyway, thanks for clarifying this. I’ll be using the ‘–no-metadata’ option in my backup script.
[. . .]
>> What would be the correct way to get rid of the snapshot, then, assuming it was not created with ‘–no-metadata’ option?
>> Is my setup in a dangerously wrong state since I’ve been removing .qcow2 and .xml snapshot files manually after the blockcommit?
No, it’s not “dangerous”, I’ve certainly seen scripts like that before, I’d just expect that you’re testing the scripts carefully before applying in production env. It’s a known issue that reverting/deleting external snapshots (especially when backing chains are involved) is super tricky and requires careful consideration. See this wiki for some discussion:
http://wiki.libvirt.org/page/I_created_an_external_snapshot%2C_but_libvirt_won%27t_let_me_delete_or_revert_to_it
And, some old notes here (the “Deleting snapshots” section): https://kashyapc.fedorapeople.org/virt/lc-2012/snapshots-handout.html
NB: There are improvements upcoming (largely thanks to Eric Blake, Peter Krempa, et al) that’ll make reverting/deleting external snapshots a bit easier.
> despite removing the snap.xml file (from /var/lib/libvirt/qemu/snapshot), virsh snapshot-list will still enumerate ‘snap’, because you’ve removed the file behind libvirt’s back.
>> I shall check, but I’m pretty sure the snapshot is not enumerated anymore by `virsh snapshot-list prod` once the .xml file is moved away and libvirtd restarted.
You’re right (just tested it) — if you _restart_ libvirtd post removing the snapshot XML, indeed,
virsh snapshot-list
does not enumerate the snapshot anymore. In my earlier hasty test, while responding to you, I didn’t restart the daemon.> The use case for snapshot-create-as command without the –no-metadata option is that it trivially allows you to _revert_ to a snapshot (in ‘internal snapshots’ case). Support for reverting to an external snapshot is in-progress upstream.
>> I was in doubt because the wiki page is about external disk snapshots and it reads “Optionally, you can also supply ‘–no-metadata’ option”, suggesting the “normal” way to do external snapshots does not necessary involves that option.
It’s likely just my poor phrasing. I wrote “Optionally” because for internal snapshots, you wouldn’t want to remove the libvirt snapshot metadata (otherwise, to be able to revert to a snapshot, you have to resort to the lower level QEMU utility (
qemu-img snapshot -l
followed byqemu-img snapshot -a
to apply a snapshot), so I didn’t want to make it sound like it’s normal in all cases.>> Anyway, thanks for clarifying this. I’ll be using the ‘–no-metadata’ option in my backup script.
PS: For future discussions on these (or any related) topics, libvirt-users@redhat.com is a nice place.