Archive for December, 2009

the right way to use disk space? virtually, of course!

Monday, December 14th, 2009

I might have mentioned agedu before, a nice tool to find your least useful and ready-to-be-deleted files real quick.

Sucks when the only files you have are rather large ones that you can’t throw out, like virtual system images which can easily become more than a few gigs heavy.

Disk is cheap you say (again) and I will protest loudly; disk is not cheap for your laptop, it is not cheap for your high-performance platter server, it is not cheap for the environment and it’s ridiculous what kind of wasteful behavior the “hey, it’s cheap” mentality promotes, not all of which relates to computers (think garbage, cars, food, wars, lives…)

Regardless, if you are using KVM there is a way to save disk space, speed up disk accesses and maybe even save the environment a little: kvm ships with a little tool called kvm-img (if you’re using QEMU then it’s qemu-img), and support for a copy-on-write storage format called QCOW2.

The qcow2 format is cool because it supports compression and encryption.

Compress your images

If you cared about disk before, you could untick the “allocate all space now” and save a couple gigs on a 10G disk image, but that wouldn’t last long and you’d hear people grumble about disk corruption and such (corruption that I have never ever seen, I might interject), but now you can compress and rebase your image. Here’s how I saved 20G on my disk:

To convert your raw image to qcow2 you would do:

kvm-img convert -c -f raw -O qcow2 $IN ${IN%.img}_base.qcow2

where $IN is your existing image and ${IN%.img}_base.qcow2 is going to be the name of your new qcow2 image. If you have NADA space left, convert into tmpfs (make sure tmpfs is mounted with sufficient size), remove the raw image and copy the new image out of tmpfs. That’ll free up some space.

Rebasing

But why stop there? I mentioned rebasing, and rebase we shall.
The qcow2 format it is a little less cool for introducing really sucky snapshotting support, as applying and creating snapshots with kvm-img takes hours and is likely to fail! I don’t recommend trying kvm-img snapshot -c foo.qcow2
However, the copy-on-write functionality of qcow2 lets us implement functional faux snapshotting with little effort.

Copy-on-write means we can create an image sliver that only stores the changes from some read-only base image. Even better, we can layer these slivers! So, with the script I’ll introduce in a second, we can:

  1. Create or convert into a compressed base image. Name it foo_base.qcow2, eg “debian_squeeze_base.qcow2″. This is the master base, ideally made right after installing the operating system or whatevr.
  2. Create a usable sliver to store new data into: kvm-img create -b debian_squeeze_base.qcow2 squeeze_today.qcow2
  3. If you are using libvirt, update your /etc/libvirt/qemu/.xml disk source file to point to the ‘today’ image, and restart the libvirt daemon and virt-manager, to catch on to the changes
  4. To create a faux snapshot, just move the today image and rebase it like in step 2.
  5. To revert a faux snapshot, just replace today’s image with the snapshot.

And here is my rebase script:

kwy@amaeth:/var/lib/libvirt/images$ cat rebase_snap.sh 
#!/bin/sh

BASE=$1
if [ ! -f $BASE ]
then
   BASE=$1.qcow2
fi
if [ ! -f $BASE ]
then
   echo "No base image $BASE"
   exit
fi
REBASE=${BASE%.qcow2}_`date +%F`.qcow2
if [ -n "$2" ] 
then
   REBASE="$2"
fi
mv $BASE $REBASE
kvm-img create -f qcow2 -b $REBASE $BASE
kvm-img info $BASE 
kvm-img info $REBASE

echo "$BASE -> $REBASE"

Advantages

  • It takes 2 seconds to rebase and restore as opposed to 1 minute vmware snapshot or 4 hours to snapshot with qcow2
  • you don’t need fancy RAID or LVM tricks
  • You save space as opposed to shitty qcow2 snapshots and raw image copies
  • you can keep several versions or patchlevels of an operating system, and several application groups on the same operating system without having to reinstall the system – you already have a base image you can use!

Caveats

The experience should be pretty stable, but there is always room to shoot yourself in the foot. Here are a couple of ways you can make it hard for yourself:

  • don’t run out of disk space – it will corrupt your open images, regardless of format
  • don’t modify a base image that another image depends upon.
    Your base image knows nothing about its children (newer snapshots and ‘today’ images), so modifying the base image will cause all its children to corrupt into weirdness. That’s why the base image is “read only” and should be named appropriately.
  • don’t go down under the stairs!
  • don’t do stuff you don’t understand!
  • don’t tell me this ain’t new, cause I know!

Lame things that suck

Saturday, December 5th, 2009

The world is a difficult place, we know.
Here’s a list of things that suck unnecessarily much:

  • Fink for OSX needs Xcode dev tools.
    Why not provide a gcc/libc-dev package? No idea. General lameness from the fink developers forces you to register at the Apple Developer Connection and download 700MB of apple crap just to install and compile source packaged software in fink. LAME.
  • Fink is not Cydia on the iPhone.
    Both are based on apt and dpkg. Both run on OSX. Pooling of efforts, everyone.
  • The very fact that you have to jailbreak an iPhone is ridiculous. Goes doubletime for Xbox and PlayStation chipping, Wii softmods and DS carding. This is vendor lockdown and should be lotted with the criminally insane – vendors don’t need to take responsibility for user-created apps, but vendoirs must not stand in the way of the software evolution.
  • Tar sands. Corruption.
  • There ain’t enough time to read all the cool web comics. Games? Don’t get me started.
  • to quote a friend and collegue, “every operating system in the world. Pick one and I will tell you how much it sucks.”