FUDcon (Fedora Users and Developers Conference) is happening in Pune, India. Plenty of interesting talks/demos/hack-fests are already scheduled. I’m giving two talks/demos. One on managing KVM hypervisor based guests via shell interface using Libvirt. The other, a working illustration of FreeIPA (with co-presenter shanks ). Will post my notes very soon.
Stay tuned.
Filed under Uncategorized
Over the weekend, for fun, I decided to tinker with the virtio ballon feature. For simplicity’s sake, I tested it with native linux kvm tool. I previously wrote about it here
If you want to get a quick refresher about virtio balloon concept, a while ago, Richard W Jones made an excellent post w/ nice illustration using images over here
Before I went ahead, I pulled the latest git, and applied a couple of patches from Sasha Levin(nlkt dev), which are on kvm mailing list, which fix the behaviour of kvm tool commands ‘stat’, ‘debug’ and a couple other things. Then I compiled the kvm tool, built the kernel bzImage, on a Fedora 16 box. Here is how it went:
First, setup a new rootfs(which will be stored under ~/.kvm-tools/default), and will mount your host file system in a read-only mode in the guest:
$ ./kvm setup default Info: Your new rootfs named default has been created. You can now start it by running 'kvm run -d default' $
Then, boot the kernel bzImage with the virtio ballon option. Once, booted, guest shell access over a serial console will be presented.
(Also, the below command line will allocate the guest memory, guest vcpu depending on the host configuration)
$ ./kvm run --balloon -k ../../arch/x86/boot/bzImage # kvm run -k ../../arch/x86/boot/bzImage -m 448 -c 4 --name guest-31362 . . . [ 1.505724] Bluetooth: BNEP filters: protocol multicast [ 1.508899] Bluetooth: HIDP (Human Interface Emulation) ver 1.2 [ 1.511511] lib80211: common routines for IEEE802.11 drivers [ 1.513885] Installing 9P2000 support [ 1.517080] Registering the dns_resolver key type [ 1.518737] Magic number: 15:85:793 [ 1.520495] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found [ 1.523497] EDD information not available. [ 1.537035] Sending DHCP requests ., OK [ 1.558038] IP-Config: Got DHCP answer from 192.168.33.1, my address is 192.168.33.15 [ 1.560257] IP-Config: Complete: [ 1.561163] device=eth0, addr=192.168.33.15, mask=255.255.255.0, gw=192.168.33.1, [ 1.563257] host=192.168.33.15, domain=, nis-domain=(none), [ 1.564861] bootserver=192.168.33.1, rootserver=0.0.0.0, rootpath= [ 1.567199] VFS: Unable to mount root fs via NFS, trying floppy. [ 1.574154] VFS: Mounted root (9p filesystem) on device 0:13. [ 1.576080] devtmpfs: mounted [ 1.578602] Freeing unused kernel memory: 512k freed [ 1.580356] Write protecting the kernel read-only data: 12288k [ 1.591640] Freeing unused kernel memory: 1916k freed [ 1.601083] Freeing unused kernel memory: 1436k freed Mounting... Starting '/bin/sh'... sh-4.2#
Get the guest current memory statistics:
$ ./kvm stat -m --all
*** Guest memory statistics ***
The amount of memory that has been swapped in (in bytes):0
The amount of memory that has been swapped out to disk (in bytes):0
The number of major page faults that have occurred:40
The number of minor page faults that have occurred:495
The amount of memory not being used for any purpose (in bytes):438673408
The total amount of memory available (in bytes):449032192
$
List the current running guest instances:
$ ./kvm list -i PID GUEST 31362 guest-31362 $
Now, on a different terminal, let’s try to inflate/deflate the balloon. The current available memory, is roughly 448MB (Refer above. In bytes — 449032192)
To inflate, which means, to return memory from the guest, run the below command, which returns 10MB of memory back to the guest.
$ ./kvm balloon -n guest-31362 --inflate 10 $
Run the memory status again. Now, 10485760 bytes(10MB) is reduced from the guest and returned to the host. (which means, the virtio balloon is inflated)
$ ./kvm stat -m --all
*** Guest memory statistics ***
The amount of memory that has been swapped in (in bytes):0
The amount of memory that has been swapped out to disk (in bytes):0
The number of major page faults that have occurred:40
The number of minor page faults that have occurred:502
The amount of memory not being used for any purpose (in bytes):428339200
The total amount of memory available (in bytes):438546432
$
Now, let’s return back the memory to the guest from the host, by running the deflate command
$ ./kvm balloon -n guest-31362 --deflate 10 $
Again, re-check the memory status. 10485760 bytes(10MB) is now returned back to the guest from the host. (which means, the virtio balloon is inflated)
$ ./kvm stat -m --all
*** Guest memory statistics ***
The amount of memory that has been swapped in (in bytes):0
The amount of memory that has been swapped out to disk (in bytes):0
The number of major page faults that have occurred:40
The number of minor page faults that have occurred:502
The amount of memory not being used for any purpose (in bytes):436846592
The total amount of memory available (in bytes):449032192
$
Now, like me, if you’re wondering what happens if you reduce the memory( inflate) by 10MB, and try to return back(deflate) 100 MB to the guest, it won’t let you.
Thanks to Sasha for helping me to fix some of the compile errors w/ newest gcc.
Filed under Uncategorized
Me and Shanks were trying to configure IPA server installation with an external CA set-up. For testing purpose, we planned to use certutil — part of mozilla nss-tools (Network Security Services) package. IPA generates a CSR(certificate signing request) called ipa.csr when you try to install it w/ the option –external-ca . The plan is to sign ipa.csr with an external CA(let’s say SubCA) which is signed by another CA(RootCA). So that there is a certificate chain of trust involved. After a bit of trial and error, this how it worked out:
To auto-generate a CSR (located /root/ipa.csr) to be signed by an external CA, while configuring IPA server, run:
$ ipa-server-install --external-ca
For illustration purpose, let’s create a temporary NSS db
$ certutil -N -d /var/tmp/testdb
Create a (self-signed)Root CA
Now, create a (self-signed)Root CA, and list the contents of the NSS(Network Security Services) db
$ certutil -S -n "IPA ROOTCA certificate" -s "cn=CAcert" -x -t "CT,," -m 1000 -v 120 -d .
$ certutil -L -d .
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
IPA ROOTCA certificate CTu,u,u
Note: Under the ‘Trust Attributes’, the ‘u’ attribute indicates that corresponding private key is also available in that database. Keys can be listed by running # certutil -K -d /path/to/nssdb
Setup Subordinate CA
Create a CSR for the Subordinate CA which will be signed by the Root CA created in the previous step
$ certutil -R -s "cn=SUB-Certificate Authority, O=LAB.TEST.REDHAT.COM" -p "9323" -o ipasubca.req -d . -a
Now, sign this CSR with Master CA and output the certificate to a file
$ certutil -C -m 2346 -i ipasubca.req -o ipasubcacert.crt -c "IPA ROOTCA certificate" -d . -a
And, add this certificate to the NSS db, and provide a nick name to it.
$ certutil -A -d . -i ipasubcacert.crt -t "CTu,Cu,Cu" -n "IPA SUBCA certificate"
Now, list the certificates. (Both certs are listed and available)
$ certutil -L -d .
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
IPA ROOTCA certificate CTu,u,u
IPA SUBCA certificate CTu,Cu,Cu
Sign certs using the SubCA
First, export the ROOTCA cert into b64 ascii cert. So that we can later create an ascii chain ca cert file
$ certutil -L -d . -n "IPA ROOTCA certificate" -a > iparootca.crt
Create a CA chain file by concatenating iparootca.crt and ipasubcacert.crt into one single file.
$ cat iparootca.crt ipasubcacert.crt > chainca.crt
Let’s sign the ipa.csr using the SubCA configured previously.
$ certutil -C -m 2346 -i ipa.csr -o ipa.crt -c "IPA SUBCA certificate" -d . -a
Now, to install IPA server successfully using the external chainca set-up, provide this command-line
$ ipa-server-install --external_cert_file=ipa.crt --external_ca_file=/tmp/chainca.crt
Voila! IPA server successfully configured. [I didn’t include the complete, long stdout of ipa-server-install to keep this brief.]
Also note: Instead of the ipa.csr, you can sign any other CSRs(user,server,etc..) generated using either certutil or any other tool(like openssl) using the above RootCA/Subca set-up.
Filed under Uncategorized
For distro upgrades, I usually use pxe/tftpboot and do clean remote installs. So I thought I’ll give yum preupgrade a try for my remote test machines. I should also mention, this F15 machine is a minimal(@core) install.
Existing F15 install
# cat /etc/redhat-release Fedora release 15 (Lovelock)
Run the pre-upgrade command to upgrade to F16 branched pre-release
preupgrade-cli "Fedora 16 Branched Pre-release (Verne)"
Everything went smooth.
Once, preupragde-cli is done, run the yum distribution-synchronization which synchronizes the installed package set with the latest packages available.
# yum distro-sync
Post preupgrade, let’s do some quick clean-up of dependency problems, orphans(installed pkgs which are not available from currently configured repos), duplicates, leaves(packages which are no longer needed by any other pkgs on the system)
# package-cleanup --problems ; package-cleanup --orphans \ package-cleanup --dupes ; package-cleanup --leaves
And reboot the host. There we go !
# cat /etc/redhat-release Fedora release 16 (Verne)
Filed under Uncategorized
Libvirt 0.9.6 was recently out. Take a look at 0.9.5 changelog for truckload of features/bugfixes/cleanups(specifically snapshot related) from the libvirt team.
So, I grabbed the F14 srpm from Libvirt ftp, and made a quick Fedora koji scratch build of libvirt-0.9.6 for Fedora 15 and gave the snapshot features a whirl. Here it goes:
(Also noted below is some very useful discussion I had(on #virt, OFTC) with Eric Blake (Upstream/Red Hat’s Libvirt dev, very friendly guy.) on snapshots. It was way informative not to capture it.)
Context on types of snapshots
At the moment, snapshotting in KVM/QEMU/Libvirt land is supported primarily for QCOW2 disk images. I briefly discussed about Qcow2 previously here.
There are several different types of snapshots possible. Some idea on that:
Internal snapshot: A type of snapshot, where a single QCOW2 file will hold both the ‘saved state’ and the ‘delta’ since that saved point. ‘Internal snapshots’ are very handy because it’s only a single file where all the snapshot info. is captured, and easy to copy/move around the machines.
External snapshot: Here, the ‘original qcow2 file’ will be in a ‘read-only’ saved state, and the new qcow2 file(which will be generated once snapshot is created) will be the delta for the changes. So, all the changes will now be written to this delta file. ‘External Snapshots’ are useful for performing backups. Also, external snapshot creates a qcow2 file with the original file as its backing image, and the backing file can be /read/ in parallel with the running qemu.
VM State: This will save the guest/domain state to a file. So, if you take a snapshot including VM state, we can then shut off that guest and use the freed up memory for other purposes on the host or for other guests. Internally this calls qemu monitor’s ‘savevm’ command. Note that this only takes care of VM state(and not disk snapshot). To try this out:
#------------------------------------------------
# Memory before saving the guest f15vm3
$ free -m
total used free shared buffers cached
Mem: 10024 5722 4301 0 164 4445
-/+ buffers/cache: 1112 8911
Swap: 0 0 0
#------------------------------------------------
$ virsh list
Id Name State
----------------------------------
5 f15guest running
6 f15vm3 running
#------------------------------------------------
# Save the guest f15vm3 to a file 'foof15vm3'
$ virsh save f15vm3 foof15vm3
Domain f15vm3 saved to foof15vm3
#------------------------------------------------
# Now, f15vm3 is gracefully saved/shutdown.
$ virsh list
Id Name State
----------------------------------
5 f15guest running
#------------------------------------------------
# Notice the RAM being freed
$ free -m
total used free shared buffers cached
Mem: 10024 5418 4605 0 164 4493
-/+ buffers/cache: 760 9263
Swap: 0 0 0
#------------------------------------------------
# Let's restore the guest back from the file 'foof15vm3'
$ virsh restore foof15vm3
Domain restored from foof15vm3
#------------------------------------------------
# List the status. f15vm3 is up and running.
$ virsh list
Id Name State
----------------------------------
5 f15guest running
7 f15vm3 running
#------------------------------------------------
For brevity, let’s try out internal disk snapshots where all the snapshot info. (like disk and VM state info) are stored in a single qcow2 file.
Virsh(libvirt shell interface to manage guests) has some neat options for snapshot supports. So, I’ve got an F15 guest (Qcow2 disk image).
Internal Disk Snapshots when the guest is online/running
For illustration purpose, let’s use a Fedora-15 guest called ‘f15guest’ .
$ virsh list Id Name State ---------------------------------- 4 f15guest running $
For clarity, ensure there are no prior snapshot instances around.
$ virsh snapshot-list f15guest Name Creation Time State ------------------------------------------------------------ $
Before creating a snapshot, we need to create a snapshot xml file with 2 simple elements (name and description) if you need sensible name for the snapshot. Note that only these two fields are user settable. Rest of the info. will be filled by Libvirt.
$ cat /var/tmp/snap1-f15guest.xml
<domainsnapshot>
<name>snap1-f15pki </name>
<description>F15 system with dogtag pki packages </description>
</domainsnapshot>
$
Eric Blake noted that, the domainsnapshot xml file is optional now for ‘snapshot-create’ if you don’t need a description for the snapshot. And if it’s okay with libvirt generating the snapshot name for us. (More on this, refer below)
Now, I’m taking a snapshot while the ‘guest’ is running live. Here, Eric noted that, especially when running/live, the more RAM the guest has, and the more active the guest is modifying that RAM, the longer the it will take to create a snapshot. This was a guest was mostly an idle guest.
$ virsh snapshot-create f15guest /var/tmp/snap1-f15guest.xml Domain snapshot snap1-f15pki created from '/var/tmp/snap1-f15guest.xml' $
While the snapshot-creation is in progress on the live guest, the state of the guest will be ‘paused’.
$ virsh list Id Name State ---------------------------------- 4 f15guest paused $
Once, the snapshot is created, list the snapshots of f15guest
$ virsh snapshot-list f15guest Name Creation Time State ------------------------------------------------------------ snap1-f15pki 2011-10-04 19:04:00 +0530 running $
Internal snapshot while the guest is offline
For fun, I created another snapshot, but after shutting down the guest. Now, the snapshot creation is just instantaneous.
$ virsh list Id Name State ---------------------------------- $ virsh snapshot-create f15guest Domain snapshot 1317757628 created
List the snapshots of ‘f15guest’ using virsh.
$ virsh snapshot-list f15guest Name Creation Time State ------------------------------------------------------------ 1317757628 2011-10-05 01:17:08 +0530 shutoff snap1-f15pki 2011-10-04 19:04:00 +0530 running
To see some information about the VM size, snapshot info:
$ qemu-img info /export/vmimgs/f15guest.qcow2 image: /export/vmimgs/f15guest.qcow2 file format: qcow2 virtual size: 8.0G (8589934592 bytes) disk size: 3.2G cluster_size: 65536 Snapshot list: ID TAG VM SIZE DATE VM CLOCK 1 snap1-f15pki 1.7G 2011-10-04 19:04:00 32:06:34.974 2 1317757628 0 2011-10-05 01:17:08 00:00:00.000 $
To revert to a particular snapshot, virsh snapshot-revert domain snapshotname
Also, discussed with Eric, in what cases does virsh invoke Qemu’s ‘savevm‘ and ‘qemu-img snapshot -c‘ commands while creating different types of snapshots discussed earlier above. Here is the outline:
(Note: –disk-only is an option to capture only ‘disk state’ but not VM state. This option is available w/ virsh ‘snapshot-create’ or ‘snapshot-create-as’ commands.)
Thanks Eric for the detail.
Filed under Uncategorized
Kashyap Chamarthy 