Posts Tagged ‘code’

pixie dust

Thursday, February 2nd, 2012

we’ve booted backtrack off usb before, now that’s kinda
boring and installing backtrack onto the usb with unetbootin
is painfully slow and not the same as bootin strait off the
usb which is what we want in this case; not an install
but a fresh copy every boot

there is someone disagreeing in the back of the room, now
wouldn’t this be a lot more complicated? No sir. on the contrary
booting fresh every time makes work a lot simpler; you gain a
direct relationship to what you store where, and where you
access your data from

but there is another one in the front;you sir, you feel that
one would have to sacrifice many of the comforts such as all
any tools of the trade at hand and permanent local storage -
but at best this is a lazy roadblock to salvation; by booting
off of local storage we have local storage at hand in a more
practical format, be that even a microscopic carrier can be
removed and replaced with sufficient storage for everything
and then some

the medium can be embedded, destroyed or ingested, so
the impermiableness of accidentally recorded data and the
robustness, accessability and portability of removable storage
comes very much in hand upon situations that either require
inconspiciousness, anonymity, covertness, plausible deniability
or a high degree of reliability in day-to-day computing

the totalality of the system given to remaining only in memory
causes it to be independent of other storage for operations, and when
operations cease from loss of any exterior preconditions, the
system simply ceases. when preconditions reoccur – by powering on
and executing the first block – the system can be relied upon to
simply starts afresh, completely unperturbed by any previous history

should the need arise to patch the system; say some new app or
capability is called for where there is no time to rebuild,
a patch should be scripted always when there is certanity that
the capability will require a repeat performance. It is advised
to devise a patch which includes all dependencies.

thus the fresh system becomes more capable and more accessible
over time, just like an install. patches can then easily be
rolled into the system should they proove useful to others.

But how does one do it? Well, it’s easy but unfortunately
not as easy as overwriting the boot device; it’s just not
practical because partitioning is always an individual consideration

  • . there are often other files on the block device
  • . choice of filesystem and memory technology has much bearing
  • . the block device is larger or smaller than expected
  • instead, we allow any bootable partition scheme and any
    filesystem and memory technology, as long as the storage
    requirements of the system are met;

    here’s to clone how:

    cp -a boot/ apt/ casper/ gone/ preseed/ syslinux/ 
    syslinux /dev/partition
    mbr /dev/device
    

    but that’s fine, it’s been done and all, but even the ability to
    boot the system with precisely zilch local storage comes in
    handy, and for that we have pixie dust.

    pixie daemon and tiny ftp should be pointing a path
    exactly matching the dhcp-provided patch.. otherwise
    you will have worries!

    /etc/pxe.conf:

    interface=eth1
    service=X86PC,0,0,local,Local boot
    service=X86PC,0,0,pxelinux,PXELinux
    tftpdbase=/var/lib/tftpboot
    domain=truly.yours
    

    /etc/default/tftpd-hpa:
    TFTP_DIRECTORY=”/var/lib/tftpboot/”

    /etc/dnsmasq.conf:

    dhcp-boot=/var/lib/tftpboot/pxelinux,vulcano,10.10.10.86
    

    “high speed” tftp daemons and multicast can be found but it is
    advised to stick to tftpd-hpa and dnsmasq with no esoterics due
    to the sheer amount of variables introduced.

    /var/lib/tftpboot/pxelinux.cfg/default:

    # not strictly necessary but makes the menu pretty
    menu hshift 13
    menu width 49
    menu margin 8
    
    menu title BackTrackBoot
    default vesamenu.c32
    display f.txt
    timeout 600
    
    label local
    menu label Local Harddisk
    localboot 0
    
    menu begin bt
    menu title BackTrack 5
    # ok here comes the real shit
    label backtrack5
    menu label BackTrack R1
    kernel bt5/vmlinuz
    append boot=casper netboot=nfs nfsroot=vulcano:/mnt/bt5 initrd=bt5/initrd.gz text splash vga=791 file=/cdrom/preseed/custom.seed --
    menu end
    

    you’ll need to copy to tftpboot/bt5 the initrd.gz and vmlinuz from the backtrack ISO /casper folder (which you can mount -o loop -t iso9660 bt5.iso /mnt/bt5

    the rest of the files you provide to the bootee over NFS

    /etc/exports:

    /mnt/bt5 10.10.3.0/24(rw,sync,no_subtree_check) 10.10.10.0/24(rw,sync,no_subtree_check)
    mount -t iso9660 -o loop BT5R1-GNOME-32.iso /mnt/bt5
    

    add a http server with kickstart / preseed files for an ever more powerful setup,
    in which case you replace the file= stanza in the append line with
    url=http://host/path/to/preseed

    more on preseeds… maybe later.

    Now restart all dependent services:

    /etc/init.d/nfs-kernel-server restart
    /etc/init.d/tftpd-hpa restart
    /etc/init.d/apache2 restart
    /etc/init.d/pxe restart
    

    debugging this setup usually requires tracing the process that is failing, so:
    - dhcp options tracing (dnsmasq verbose and tcpdump / wireshark)
    - verbose pxe
    - verbose foreground tftpd-hpa : in.tftpd -v -v -L /var/lib/tftpboot

    CPM 0.26 the Console Password Manager

    Monday, December 5th, 2011

    Some of you might have noticed that I’ve adopted this little program while its original author is MIA, and that my efforts have resulted in its inclusion into debian wheezy earlier this year.

    This is great news and makes it a breeze to get up and running with CPM with a simple apt-get install cpm

    However, it seems that most people are interested in running CPM on older distributions, realistically the stable distribution codenamed squeeze is a favorite, as well as the Ubuntu LTS release 10.4 codenamed lucid lynx.

    So I have built some updated packages of CPM for these oldies but goodies:
    * CPM for squeeze i386
    * CPM for squeeze amd64
    * CPM for lucid i386
    * CPM for lucid amd64

    Remember to install the dependencies though. On squeeze, they are:

    me@mine:~# apt-get install \
        libcdk5 libcrack2 libdotconf1.0 libgpg-error0 \
        libgpgme11 libxml2 libxml2-utils libpth20
    

    File us a ticket if you run into trouble with these packages or need cpm working on some other distribution.

    CPM is a simple, paranoid password manager for the console with some cool features that make it stand out:

    * data files can be encrypted for more than one person
    * data files are signed by the last person who saved it so forging data files is not possible
    * data files are en- and decryptable directly by gpg and gzip
    * the application memory is protected from paging, core dumps, ptrace attacks and runtime environment
    * data is validated using an internal DTD
    * several passwords per account are possible to store
    * it’s possible to handle several data files, each encrypted for different people
    * cracklib checks of password strength and warnings about weak passwords
    * user definable hierarchy with unlimited depth
    * long comments for any node in the hierarchy
    * password generator
    * only one password visible at a time
    * searchable database from the command line
    * user definable search patterns (e.g. user@hostname)
    * several hits can be displayed at once (e.g. several accounts per host)
    * conversion scripts for Password Management System (pms), Password Safe and CSV files

    failcode

    Thursday, August 18th, 2011

    In my time as an application programmer. developer and designer, breif stint as team lead and project manager,
    as well as my time as a systems consultant, I have witnessed first-hand and also heard many credible tales of systematic failure that rival any of the stories on The Daily WTF. My collegues and I have seen so many examples of bad design, bad code and systemic failure that we have considered writing a book titled How To Write Ugly Code.

    I have also read the Texas Instruments Chainsaw massacre and personally met Gomez while debugging applications.

    My speciality and my interest lies in diagnostics and the analysis of problems as well as system security, and my experience has showed that one can venture to say something about the qualitative difference of different design methodologies and what they have to say for the end result.

    Firstly however, it is worth noting that the software industry as a whole has one primary problem: the time pressure to deliver new features at the face of mouting expectations.

    This pressure to deliver is seen as the driving force behind industry progress and ever leaner, more economic applications, however contrary to this belief I have evidence that it leads to incentives for sloppy work, overengineering and poor considerations of the problem domain. It seems the process itself rewards poor application design, regardless of development methodology.

    Large corporate and government tenders, which affect many hundreds of thousands of peoples lives, get bid on by large software houses that believe they can deliver everything (at low risk: if they cannot deliver it is very hard for the customer to contest this to a major software house).

    What we get by and large out of this process are bloated top-down applications designed by people who do not understand the (whole) problem, leading to misguided decisions for such things as

    • choice of platform and language
    • choice of coding standards (check out Systems Hungarian if you don’t believe me)
    • programming methodology
    • communication tools: source control, ticket and forum tools for developers and system managers
    • Not Invented Here-practices
    • monkey-coding by people whose talents could be employed to solving the problem

    What usually goes for as “agile” development causes frequent ineffective blame-game meetings.
    Unit test driven development frequently causes micromanagement of program details and inflexible designs,
    … all these methodologies were designed to improve programs, not bog them down! why then do they cause so much breakage?

    The pressure to deliver requires the application developer to prefer large swathes of ready-made library code and a high level of abstraction to allow her to meet deadline demands.

    A high abstraction level causes low debuggability and poor performance.
    Low debuggability because bugs are by definition conditions caused by circumstances unforseen by the application developer. Abstractions are employed by the developer to hide implementation details to aid clairty and speed of application development, at the cost of debuggability.

    The very tools and abstractions employed by the application developer create the frame through which the developer can see the circumstances of her design and code. Bugs most frequently occur on the boundries between abstractions, where the developer has no possibility to forsee these circumstances. Furthermore, in a system that has a passibly high level of abstraction there is a whole stack of hidden details which must be traced and unwound to discover the bug. Therefore, every additional layer of abstraction obscures the debugging process.

    The debuggability and algorithmic simplicity is key in achieving optimal performance. In other words, if we have a clear problem statement it is possible to achieve performance. If there is no clear problem statement, and the program is further muddled by abstractions and interactions there is no effective path to performance.

    Any artist will be able to tell you that the most interesting, creative and innovative work comes out of having a stress-free, playful environ. Since innovative coding is a creative activity, the same applies to developing applications, something that game developer companies and creative shops have known for years, and behemoths like Google and Microsoft have picked up on, reinvesting up to 15% of their revenue into research and development and getting that part right, as witnessed by the sheer output of innovation.

    If there is a clear path to solving these fundamental problems of IT then it is putting the people who know what they are doing in the pilot seat, enabling developers to choose for themselves not only toolchains, methodology and communication tools but also engaging the systems thinkers into creating the specifications and architecture of the systems they are going to implement. The good news is that as customers and managers get savvy to this method of achieving IT success, we are going to see more developer autonomy and less spectacular fails.

    consolekit is evil

    Wednesday, December 1st, 2010

    … and hates me

    I should really tell you about the DLD seminar three weeks ago, or the PARANOIA security conference, or even just that Adobe should be considered harmful but things have been crazy and between this and electromagnetism I haven’t had the mind space. After the 6th of december, I promise I’l come back with pictures and relations and maybe even sounds (I have notes, don’t worry I’ll remember).

    On the other hand here’s a nasty hack to kill console-kit-daemon, which has a really nasty way of polluting the PID-space… and annoys me enough to warrant a public humiliation as well. What does it do, and why? Who cares what it does, it’s doing it poorly enough to catch attention to itself! So here’s how to kill it:

    root@wasp:/usr/sbin# dpkg -S console-kit-daemon
    consolekit: /usr/sbin/console-kit-daemon
    

    DON’T try to purge the package because that’s just one end of a really big ugly yarn of unneccessary dependency pain that I’d like to spare you…

    DON’T try to replace /usr/sbin/console-kit-daemon with your own stub… turns out dbus autostarts this “service”, and that approach will make dbus block your (ssh) session when you log in… not forever, but that’s even more annoying than the pid pollution.

    Instead, debian bug #544147 and #544483 clewed me in to the following hack:

    cp /usr/share/dbus-1/system-services/org.freedesktop.ConsoleKit.service \
      /usr/local/share/dbus-1/system-services/
    echo Exec=/bin/false >> /usr/local/share/dbus-1/system-services/org.freedesktop.ConsoleKit.service
    

    which is a two-liner, and would have been less ugly and easier to debug if it hadn’t been for the fine hubris of the freedesktop dudes…

    OSSEC to the rescue

    Wednesday, October 20th, 2010

    I’m not going to brag about being online for 16 years without being hacked. It simply wouldn’t be truthful and more to the point even if I convinced myself there is little you or I can do to verify the claim. Rather, I’d like to think that by being a little paranoid I’ve managed to avoid some badness. Actually even if you like to think so, it’s rather optimistic to believe in one’s own infallability. The infallability of computer systems? Don’t even get me started.

    Computer security is about turning that trend around, about saying OK where and how did the bad guy get in, lets kick him out and make sure it doesn’t happen again. It’s about fixing the problems before they become really bad. Security is also about not putting all your balls in one basket, not trusting your single point of failure, and being very picky about the things you trust. Because automated systems fail automatically, security is about putting youself in the loop too.

    If you haven’t read this year’s Verizon data breach report [1], the gist is basically that 2/3 hacks are from 3rd parties, that leakage usually occurs 3 minutes into the breach and that most attacks are still discovered by a third party. What more, almost all attacks were made on servers, most of the attacks aren’t even that difficult to do, they leave traces in the log files and the security holes are even easier to fix!

    Now if you’ve been paying attention to the Stuxnet infestation [2], the Microsoft hack[3] or the recent Depnet failure[4], there is no guarantee that your skilled and educated IT staff is on top of that stuff… because they’re too busy delivering the features you demand.

    The problem here is one of control. If you are an admin, you may know what you’ve done on any particular server and you might be on top of what your team has been doing, but the second someone gets in and starts changing things they shouldn’t have, the bets are off. Files get changed: logs get nuked, commands get replaced, databases get UPDATE’d.

    Let me tell it to you straight: a virus, worm, piece of malware is basically a really cool software update.

    What you need is an eventuality: something that leaves a central, verifiable audit log, checks a number of different sources, stores who logged in where how and monitors system integrity. You need something flexible, with 90% of the job already done for you, something that can deal with a bunch of computers on a bunch of different platforms at the same time, and while leaving you in the loop does this in a managable way, so you don’t get overblown by a zillion messages.

    You need something that can respond to its environment, something that sits on every host, something that can take action on its own.

    OSSEC[5] has this three-pronged approach that fits the bill: rootkit checks, file integrity and logfile watching.

    It does these things according to a configurable ruleset. The rules can run on all your computers (yup, windows boxes too) and report to a central OSSEC server securely. OSSEC is also able to respond to alerts, for example by blocking an attacker that is trying to guess the password repeatedly (an everyday occurance).

    What more, GPL open source makes it possible to audit and patch the code of OSSEC, and gracefully sidesteps the problem of vendor lock-in.

    Now that I’ve played with it and tuned it for sufficiently long, it’s started to compliment my IDS nicely and beats old approaches like tripwire, fail2ban[6]/sshguard[7] and logwatch[8]. Don’t get me wrong, OSSEC is not the silver bullet[9], then again nothing is and thus we must stay vigilant.

    So, with the advent of Week of OSSEC year 2 I took the opportunity to tell you about this fine piece of software, and to show you the real nugget: my debian install and update script for ossec which you can use standalone, or together with my budding and passably simple configuration system gone, which I will introduce another day in another post.

    0K out.

    References in all their undistractingly subscripted glory:
    [1] Verizon data breach report
    [2] Talk on stuxnet the SCADA worm by kwy
    [3] Microsoft confirms Russian pill-pusher attack on its network
    [4] Regjeringen utsatt for dataspionasje
    [5] OSSEC
    [6] Fail2ban
    [7] SSHguard
    [8] Logwatch
    [9] Abusing OSSEC

    CPM 0.25 :: new packages

    Wednesday, September 15th, 2010

    sup peeps,
    your favorite password managment program, CPM, now has updated packages.
    What’s new in these packages is a working create-cpmdb. A fix for setting the SUID bit is also included, and that will allow CPM to store passwords securely in memory as well as on disk.

    Ubuntu and Debian users:
    Get cpm_0.25~beta-2debian3_amd64.deb directly from github.

    I have also taken the opportunity to update the documentation, which will allow you to roll your own CPM should you be running something other than debian.

    Quick start:

    me@mine:~# apt-get install libcdk5 libcrack2 libdotconf1.0 libgpg-error0 libgpgme11 libncursesw5 libxml2 libxml2-utils zlib1g
    me@mine:~# dpkg -i cpm_0.25~beta-2debian3_amd64.deb
    

    You need: a GPG key and 3 minutes of your time. Create the password database (only once):

    me@mine:~$ create-cpmdb
    

    Use your GPG key to encrypt the database. This puts a .cpmdb file in your home folder.

    Run CPM and add your passwords! Exit by hitting ESC to save the keys.

    me@mine:~$ cpm
    

    and you have a working CPM install.

    Furthermore, I have devised a way for many people to share the same passoword database through a revision control system. Take a look at CPM::revision control.

    backtrack to install a backtrack

    Thursday, September 9th, 2010

    BackTrack is your daddy.
    BackTrack accepts no compromises, yet it is all compromising.
    Because really, when is the last time you *didn’t* need those auditing tools? That penetration suite? Total privacy to break other people’s privacy? All that and a packet of crisps wrapped with razor sharp menus – it’s the kind of stuff you can only dream of on core. And I hear Fedora Core is the shitzitz now, adopting new [1] and exciting[2] features. Oh hey debian doesn’t have binary deltas for packages *yet* [3], but we’ve been talking about it way longer than those dudes have.

    Anecdtotally, I spilled a glass of water on my laptop the other day. Naturally, the glass went half empty in an instant: my poor lovely x41, I screamed. As it turns out the laptop casing made sure all the water was rather cleverly funneled into the x41′s only 1.8″ harddrive, which proceeded to go completely bananas (due presumably to rust, because clean water doesn’t conduct, right?). The data? I believe trusty old dd_rescue did rescue at least part of it, but I then misplaced the image file somewhere.

    The system?
    It was a thrifty, untrusted yet trusty Windows XP install that I’d been keeping on there on the mercy of actually booting every time since I bought the machine despite having been licked by more than its fair share of virii, malignant updates and accidental hard resets. Most of the programs I ran were portable[4] versions so all I lost were some documents and lots of music[5].

    The hardware?
    I disassembled and metricuously dried every little component, and in the end only the disk drive was bust. The 1.8″ IDE drive that is impossibly ridiculously expensive to replace (5$ per GB? What the foo? Shut up!). Still, I needed the laptop so I exploded booting from USB. Despite (misguided?) efforts I haven’t bloody well been able to boot windows off USB, so I bootstrapped BackTrack 3 instead and bob is your uncle.

    I mean really, I think I had that thing running like that for three months before I started missing stuff like apt. Didn’t really mind starting fresh every boot, I even invented a whole little schpiel for getting online as fast as possible, none of that Network Manager madness.
    Persistent settings are all right in BT3 but booting into RAM is a lot more fun. After the first 3 seconds of boot you can pull the USB plug, everything goes zippety fast and your footprint is nada. Only thing that can get your ass is a cold boot attack.

    BT3 is real cool and still a good recommend if you want to wardrive and do proper wifi phreaking due to the embedded injection drivers, but in the end I wanted new libs, a decent compiler and window dressing, and so I rolled BackTrack 4.

    Granted, kde sucks, but if I cared enough I’d switch to openbox or something awesome in like 4 minutes. These days all I need is a shell and a browser.

    For those of you fortunate enough to have a harddrive, BT4 ships with an install script to turn your system into a permanent BackTrack fixture. It’s based off Ubiquity, but dd’ing off the USB and onto your disk drive might be better if you’re interested in being able to boot your system into RAM, well I dunno because you want to do some advanced powersaving[6], or want to kill your system without worrying about unclean shutdowns, or want to maximise the life span of your solid-state device by nearly never writing to it.

    For my own part there was a happy ending on DealExtreme, as they ship IDE44 to CompactFlash interfaces that fit in the x41 1.8″ bay… which leads to a whole slew of unexplored possibilities thaaat (drum rolls) I will explore in the next installment of how to break your machine.

    BackTrack 4 R1 has released :-) [6]. Anyone know where I can score the BlackHat Edition?

    [1] http://fedoraproject.org/wiki/Releases/FeaturePresto
    [2] http://fedoraproject.org/wiki/Features/systemd
    [3] http://samba.anu.edu.au/rsync/rsync-and-debian/rsync-and-debian.html
    [4] http://portableapps.com/
    [5] http://drownedinsound.com/community/boards/music/4179554
    [6] http://kacper.blog.linpro.no/archives/13
    [7] http://www.backtrack-linux.org/

    tune2fs and green disks

    Thursday, August 5th, 2010

    Hey folks,
    old news I’m sure, but if you get tempted into buying the new WD Caviar “Green Power” disks there is something you need to know about them: they fake 512-byte blocksizes while in reality having 4096-byte blocks! The move to 4K blocks is reasonable considering we just busted the 2 terabyte barrier, but the disk firmware is faking 512-byte blocks in the name of compatibility (read: so windows xp won’t shit itself).

    Unfortunately, running in bs512 mode makes the disk exactly 3x slower than it should be!
    The fix: line up your partitions at 4k boundries, so start partition one at block 64, 1024 or even 2048 (the win7 start block) not the default, 63, in most partitioning software. Start fdisk with the -u parameter and carefully specify the start block. In gparted you’ll have to unhook the “snap to cylinder boundries” checkbox, and then I suppose you could even move a partition to the right block, but expect this to take an inordinate amount of time!

    On a related note, fsck’ing an ext filesystem on boot is a drag, and fsck’ing 2TB file systems is a huge drag. Sure you should be running the fsck but it has a nasty tendency to happen on your workstation precisely when you can’t afford the extra 5 minute delay!

    I bump the default 10 mounts count to 0 (disabling mount count fscking) and auto-fsck my disks every 99 days, staggered so not all disks get checked on the same day. Do this with the tune2fs command:

    wasp:~# tune2fs -c 0 -i 99d /dev/sda1
    tune2fs 1.41.12 (17-May-2010)
    Setting maximal mount count to -1
    Setting interval between checks to 8553600 seconds
    

    out.

    PS I recently managed to achieve sustained throughputs of 110MB/s with these WD disks and properly aligned partitions:

    7516192768 bytes (7.5 GB) copied, 68.4392 s, 110 MB/s
    115+0 records in
    114+0 records out
    

    yes that’s disk-to-disk with ext4 and one large file, no fragmentation.

    PPS the defaults have nowadays changed to 120 days and 39 mounts, to which I say -1 mounts is better anyway!

    edit: Now that your files are aligned, you can specify a block size to mkfs as well, which might avoid unaligned fragments: mkfs.ext4 -b 4096 -L gigantor -O sparse_super /dev/sdb1

    EDD DoS detection and DLD

    Friday, April 16th, 2010

    Hi all,
    a short note about the Norwegian data surveillance directive that is up for passage into law these days. This directive, “Datalagringsdirektivet”, is the single most harmful threat to the general public’s privacy while being completely ineffective at stopping the bad guys it’s meant to target. Protests last Saturday in front of the parlimentary building – which yours truly attended – featured politicians and individuals from all ends of the political spectrum. Read more on Stopp DLD.

    On to other things,

    getting DDoSed sucks, as some of my collegues found out recently. Wouldn’t it be great if we could detect DDoSes as they come in through the wire? I mean besides when all of nagios goes code red upon us?

    Well, I’ve written the little program that could. It’s not quite there yet (too few hours in the day) but the basic principles are fleshed out, and they go a little something like this:

    There is a mathematical and a physical notion of entropy. To put it bluntly, it’s the shortest representation of a given piece of information. There’s a theorem that states that if you get many messages, but the messages put together don’t amount to much, then probably someone is fugging with you. We can use this to detect anomalies in network traffic, too.

    This theorem about entropy is what EDD, the Entropy Distributed Denial of Service Detector [tarball] uses to classify a packet stream as bollocks, or not.

    EDD is still pre-alfa software, which means that it’s a little too simplistic to tell you anything beyond a mere “Something’s up”, but I’d like you to test it in your setups with the understanding that the program.

    commit bc2f4df34745e4c422a17e70aac271bc930b9f1a
    Author: Kacper Wysocki
    Date: Fri Apr 16 18:18:37 2010 +0200

    EDD now classifies simple SYN floods successfully.

    * faster and simpler simple_entropy
    * reads from pcaps (-r)
    * configurable treshold (-e)
    * configurable window size (-w)
    * profile counting (-t)
    * edd self-tests (-E)
    * better TODO ideas

    Try it out and let me now, and send me pcaps of your DDoS and false positives.

    the right way to use disk space? virtually, of course!

    Monday, December 14th, 2009

    I might have mentioned agedu before, a nice tool to find your least useful and ready-to-be-deleted files real quick.

    Sucks when the only files you have are rather large ones that you can’t throw out, like virtual system images which can easily become more than a few gigs heavy.

    Disk is cheap you say (again) and I will protest loudly; disk is not cheap for your laptop, it is not cheap for your high-performance platter server, it is not cheap for the environment and it’s ridiculous what kind of wasteful behavior the “hey, it’s cheap” mentality promotes, not all of which relates to computers (think garbage, cars, food, wars, lives…)

    Regardless, if you are using KVM there is a way to save disk space, speed up disk accesses and maybe even save the environment a little: kvm ships with a little tool called kvm-img (if you’re using QEMU then it’s qemu-img), and support for a copy-on-write storage format called QCOW2.

    The qcow2 format is cool because it supports compression and encryption.

    Compress your images

    If you cared about disk before, you could untick the “allocate all space now” and save a couple gigs on a 10G disk image, but that wouldn’t last long and you’d hear people grumble about disk corruption and such (corruption that I have never ever seen, I might interject), but now you can compress and rebase your image. Here’s how I saved 20G on my disk:

    To convert your raw image to qcow2 you would do:

    kvm-img convert -c -f raw -O qcow2 $IN ${IN%.img}_base.qcow2
    

    where $IN is your existing image and ${IN%.img}_base.qcow2 is going to be the name of your new qcow2 image. If you have NADA space left, convert into tmpfs (make sure tmpfs is mounted with sufficient size), remove the raw image and copy the new image out of tmpfs. That’ll free up some space.

    Rebasing

    But why stop there? I mentioned rebasing, and rebase we shall.
    The qcow2 format it is a little less cool for introducing really sucky snapshotting support, as applying and creating snapshots with kvm-img takes hours and is likely to fail! I don’t recommend trying kvm-img snapshot -c foo.qcow2
    However, the copy-on-write functionality of qcow2 lets us implement functional faux snapshotting with little effort.

    Copy-on-write means we can create an image sliver that only stores the changes from some read-only base image. Even better, we can layer these slivers! So, with the script I’ll introduce in a second, we can:

    1. Create or convert into a compressed base image. Name it foo_base.qcow2, eg “debian_squeeze_base.qcow2″. This is the master base, ideally made right after installing the operating system or whatevr.
    2. Create a usable sliver to store new data into: kvm-img create -b debian_squeeze_base.qcow2 squeeze_today.qcow2
    3. If you are using libvirt, update your /etc/libvirt/qemu/.xml disk source file to point to the ‘today’ image, and restart the libvirt daemon and virt-manager, to catch on to the changes
    4. To create a faux snapshot, just move the today image and rebase it like in step 2.
    5. To revert a faux snapshot, just replace today’s image with the snapshot.

    And here is my rebase script:

    kwy@amaeth:/var/lib/libvirt/images$ cat rebase_snap.sh 
    #!/bin/sh
    
    BASE=$1
    if [ ! -f $BASE ]
    then
       BASE=$1.qcow2
    fi
    if [ ! -f $BASE ]
    then
       echo "No base image $BASE"
       exit
    fi
    REBASE=${BASE%.qcow2}_`date +%F`.qcow2
    if [ -n "$2" ] 
    then
       REBASE="$2"
    fi
    mv $BASE $REBASE
    kvm-img create -f qcow2 -b $REBASE $BASE
    kvm-img info $BASE 
    kvm-img info $REBASE
    
    echo "$BASE -> $REBASE"
    

    Advantages

    • It takes 2 seconds to rebase and restore as opposed to 1 minute vmware snapshot or 4 hours to snapshot with qcow2
    • you don’t need fancy RAID or LVM tricks
    • You save space as opposed to shitty qcow2 snapshots and raw image copies
    • you can keep several versions or patchlevels of an operating system, and several application groups on the same operating system without having to reinstall the system – you already have a base image you can use!

    Caveats

    The experience should be pretty stable, but there is always room to shoot yourself in the foot. Here are a couple of ways you can make it hard for yourself:

    • don’t run out of disk space – it will corrupt your open images, regardless of format
    • don’t modify a base image that another image depends upon.
      Your base image knows nothing about its children (newer snapshots and ‘today’ images), so modifying the base image will cause all its children to corrupt into weirdness. That’s why the base image is “read only” and should be named appropriately.
    • don’t go down under the stairs!
    • don’t do stuff you don’t understand!
    • don’t tell me this ain’t new, cause I know!