Live backup with external disk snapshots and libvirt’s blockpull

There’s been some interesting work going on in blocklayer for libvirt/qemu/kvm. I have a few pending posts related to that, thought I’ll start with a post about live backup with external disk snapshots & using libvirt’s blockpull. External snapshots were previously discussed here.

[1] List the block device associated with a guest.


# virsh domblklist f17-base
Target     Source
---------------------------------------------
vda        /export/vmimages/f17-base.qcow2

#

[2] Create external disk-only snapshot (while the guest is running).


# virsh snapshot-create-as --domain f17-base \
  snap1 snap1-desc --disk-only \
  --diskspec vda,snapshot=external,file=/export/vmimages/sn1-of-f17-base.qcow2 \              
  -- atomic \
 Domain snapshot snap1 created
#

[3] Now, list the block device associated(use cmd from step-1, above)
with the guest, to ensure it reflects the new overlay image as
the current block device in use.


# virsh domblklist f17-base
Target     Source
----------------------------------------------------
vda        /export/vmimages/sn1-of-f17-base.qcow2

#

[4] Here, backup the original disk image /export/vmimages/f17-base.qcow2
using rsync or any other preferred method, while the guest is now
running ( &amp writing all its new changes into sn1-of-f17-base.qcow2).

[5] Now, let’s say, there are changes accumulated over time into
sn1-of-f17-base.qcow2(current active layer). We can now merge
the contents from original disk image(f17-base) into the current disk using ‘blockpull’.

Two possible here:

(a) If we had the below chain:

 [base] <-- [sn1] # the current active layer

We can make sn1-of-f17-base.qcow2 a standalone image by
pulling/merging contents from the orignial disk image (f17-base.qcow2). Resulting in sn1-of-f17-base becoming a self-contained disk image without any backing file.


# virsh blockpull --domain f17-base --path \
  /var/lib/libvirt/images/sn1-of-f17-base.qcow2 --verbose --wait

The resultant chain will be:

 [sn1]  

which is an active, self-contained image with all the data from [base] pulled into it.

(b) If we have more snapshots:


Say, we start with this chain,

  [base] <-- [sn1] <-- [sn2] <-- [active],

it can be shortened to:

  [base] <-- [sn1] <-- [active]

by doing:

  # virsh blockpull --domain f17-base --path \
    /var/lib/libvirt/images/active.qcow2 \
    --base /var/lib/libvirt/sn1.qcow2 --verbose --wait

The above command pulls the data from [sn2] into [active], and changes the backing file of [active] to [sn1].

As an optional next step, we can again perform 'blockpull' operation,
with the above shortened chain, resulting in [active] becoming a standalone image:


# virsh blockpull --domain f17-base --path \
  /var/lib/libvirt/images/active.qcow2 --verbose --wait

The above command pulls data from [sn1] and [base]
into [active] and makes it a self-contained image without any backing file,
resulting in the chain with a single image: [active]

NOTE: 'blockpull' needs to be done for every disk image that was
snapshotted (by optionally supplying –base. If a –base option
is not supplied, the current disk image will pull data from its
entire backing chain, & makes it standalone without any backing
file.)

[6] Once the 'blockpull' operation above is complete, we can clean-up
the tracking of snapshots(metadata) by libvirt to reflect the new reality:

              
# virsh snapshot-delete $dom $name --metadata

There are couple of other ways doing guest live backups using Libvirt’s newer capabilities like ‘blockcopy’ mechanism. An unpolished example is here (This test was was done by building libvirt & qemu from git, a couple of months ago. I haven’t yet gotten to testing newest qemu/libvirt)

Advertisements

9 Comments

Filed under Uncategorized

9 responses to “Live backup with external disk snapshots and libvirt’s blockpull

  1. Tushar

    Nice post! If the qcow2 image is a bootable image (as opposed to just a block device for guest storage), then, in that case, can one use the base (read-only) copy as something that one can boot a new guest instance off?

    • Sure, it should be possible. And, the base has to be read-only. As a rule of thumb, the only editable image is the leaf image (i.e. the current active layer).

      More elaborate notes here and some more discussion on blockpull.

      Hope that clarifies.

      • Tushar

        Thanks. The question really is if we will be able to recover from such a copy. Let me explain.

        I have a live VM image backing up my guest called MyLiveVMImage.qcow2 and is present on the local (server or laptop) disk/file system.

        I create another qcow2 image called MyVirtualDiskForApp.qcow2 and export that into the guest as Virtual Block Device. This virtual block device is used by My Application running inside the guest.

        Now, for disaster recovery purposes, say, I want to write a script that hourly snaps the two qcow2 images and backs it up externally (say on S3 or elsewhere).

        Can I do the following:

        Step 1: virsh snapshot of MyLiveVMImage.qcow2
        A read only MyLiveVMImage base is created.

        Step 2: virsh snapshot MyVirtualDiskForApp.qcow2
        A read only MyVirtualDiskForApp base is created.

        Step 3:
        Copy MyLiveVMImage base and MyVirtualDiskForApp base off to a remote safe haven.

        Step 4:
        blockpull snap of MyLiveVMImage.qcow2 so that there is only one active base.

        blockpull snap of MyVirtualDiskForApp.qcow2 so that there is only one active base.

        Step 5: Go back to and repeat from Step 1 every hour.

        I am thinking this should work although my main worry would be is that since the snap of MyVirtualDiskForApp.qcow2 and MyLiveVMImage.qcow2 are not atomic, can the application really recover from my remote copies if my laptop or server indeed becomes unavailable.

      • “I have a live VM image backing up my guest called MyLiveVMImage.qcow2” -> This sentence is not clear, I interpret it as: you have a live VM with a disk image named MyLiveVMImage.qcow2.

        Again, ‘snapshot’ is a vauge word. Do you also intend to capture memory state in this case of snapshot? If yes, you might want to take a look at an example of external system checkpoint snapshot.

        I haven’t really tried the use-case you’ve outlined here. A proof-of-concept test should be tried.

        That said, couple of thoughts:

        – About atomic, you can pass --atomic flag to virsh snapshot-create-as, so that libvirt will ensure either the snapshot succeeds or fails.

        – There are more efficient ways to do live snapshots than what’s outlined in this post. For instance,
        – You might want to explore combination blockcopy, blockpull blockcommit

        Also, please note — if you create a disk-only snapshot, blockcopy --shallow is useful to copy just the delta to shared storage, so you can avoid copying/backing-up the full disks.

  2. gunnar

    not sure whether I am besides the point because I don’t understand something basic … but to me this looks like you can only start removing (external) snapshots from the wrong end so to say. An like that I don’t see any benefit in them as a solution for backing up an image. Maybe there are other awesome things you can do with them (for example running several different machines with one base only)

    Let’s say you have several states of an image a VM you have stored

    base
    -3
    -2
    -1
    -0

    (‘-3’ is the oldest ‘-0’ the current state in this example)
    You might have a policy that backups are kept until -2 and everything older then that is being removed. As far as I see through “external snapshots” so far this is just not possible, because -2 can’t run without -3 (even if all data of -3 is available in base after -3 had been deleted [and merged into base]), right?

    So, as far as I see it the big benefit of external snapshots being relatively small (and therefore suited to be backed up) can not be taken advantage of really … Or is there a way to enable -2 to run on base if -3 is merged to it? I tested this once and it failed, but I am far from being an expert. I guess -2 needed to see the ‘whole picture of ‘base’ & ‘-3’. Not sure whether there is a way to achieve that

    • To your specific question, yes, there’s a way to make [-2] to point to [base].

      Let’s represent your situation like this:

       [base] <--  [-3] <-- [-2] <-- [-1] <--[0] (currently active) 

      Since these are ‘chained’ images, you can probably use scripts that use blockpull to pull data from [-3] into base:

       $ virsh blockpull --domain myvm --path /export/images/[-2].qcow2 --base /export/images/[base].qcow2  --wait --verbose 

      Now you can discard [-3], since the above will change the pointer of [-2] to [base]. Hope that makes sense.

      Honestly, I’m not really a backups expert. But given these tools and some careful planning, there are plenty of ways one can use various policies to optimize backups of disk image chains.

      For some more understanding, I wrote some examples of blockpull with more commentary here (slightly dated, but still relevant): https://kashyapc.fedorapeople.org/virt/lc-2012/snapshots-handout.html

      From my testing, using the blockcommit approach is a bit more faster than using blockpull. Maybe you can take a look here — http://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit

      And, much lengthier topics like these are better discussed on libvirt-users@redhat.com mailing list.

  3. gunnar

    seems we have 2 different ways to merge data blockpull vs blockcommit. If I am not mistaken the difference is:

    blockpull > forward in time
    blockpull > pulls older data to newer images (external sn)
    blockpull > increases size of newer images (external sn)

    blockcommit > backward in time
    blockcommit > merges new data into older images(base or external sn)
    blockcommit > increases size of older images

    is that so?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s