Posts Tagged ‘script’

pixie dust

Thursday, February 2nd, 2012

we’ve booted backtrack off usb before, now that’s kinda
boring and installing backtrack onto the usb with unetbootin
is painfully slow and not the same as bootin strait off the
usb which is what we want in this case; not an install
but a fresh copy every boot

there is someone disagreeing in the back of the room, now
wouldn’t this be a lot more complicated? No sir. on the contrary
booting fresh every time makes work a lot simpler; you gain a
direct relationship to what you store where, and where you
access your data from

but there is another one in the front;you sir, you feel that
one would have to sacrifice many of the comforts such as all
any tools of the trade at hand and permanent local storage -
but at best this is a lazy roadblock to salvation; by booting
off of local storage we have local storage at hand in a more
practical format, be that even a microscopic carrier can be
removed and replaced with sufficient storage for everything
and then some

the medium can be embedded, destroyed or ingested, so
the impermiableness of accidentally recorded data and the
robustness, accessability and portability of removable storage
comes very much in hand upon situations that either require
inconspiciousness, anonymity, covertness, plausible deniability
or a high degree of reliability in day-to-day computing

the totalality of the system given to remaining only in memory
causes it to be independent of other storage for operations, and when
operations cease from loss of any exterior preconditions, the
system simply ceases. when preconditions reoccur – by powering on
and executing the first block – the system can be relied upon to
simply starts afresh, completely unperturbed by any previous history

should the need arise to patch the system; say some new app or
capability is called for where there is no time to rebuild,
a patch should be scripted always when there is certanity that
the capability will require a repeat performance. It is advised
to devise a patch which includes all dependencies.

thus the fresh system becomes more capable and more accessible
over time, just like an install. patches can then easily be
rolled into the system should they proove useful to others.

But how does one do it? Well, it’s easy but unfortunately
not as easy as overwriting the boot device; it’s just not
practical because partitioning is always an individual consideration

  • . there are often other files on the block device
  • . choice of filesystem and memory technology has much bearing
  • . the block device is larger or smaller than expected
  • instead, we allow any bootable partition scheme and any
    filesystem and memory technology, as long as the storage
    requirements of the system are met;

    here’s to clone how:

    cp -a boot/ apt/ casper/ gone/ preseed/ syslinux/ 
    syslinux /dev/partition
    mbr /dev/device
    

    but that’s fine, it’s been done and all, but even the ability to
    boot the system with precisely zilch local storage comes in
    handy, and for that we have pixie dust.

    pixie daemon and tiny ftp should be pointing a path
    exactly matching the dhcp-provided patch.. otherwise
    you will have worries!

    /etc/pxe.conf:

    interface=eth1
    service=X86PC,0,0,local,Local boot
    service=X86PC,0,0,pxelinux,PXELinux
    tftpdbase=/var/lib/tftpboot
    domain=truly.yours
    

    /etc/default/tftpd-hpa:
    TFTP_DIRECTORY=”/var/lib/tftpboot/”

    /etc/dnsmasq.conf:

    dhcp-boot=/var/lib/tftpboot/pxelinux,vulcano,10.10.10.86
    

    “high speed” tftp daemons and multicast can be found but it is
    advised to stick to tftpd-hpa and dnsmasq with no esoterics due
    to the sheer amount of variables introduced.

    /var/lib/tftpboot/pxelinux.cfg/default:

    # not strictly necessary but makes the menu pretty
    menu hshift 13
    menu width 49
    menu margin 8
    
    menu title BackTrackBoot
    default vesamenu.c32
    display f.txt
    timeout 600
    
    label local
    menu label Local Harddisk
    localboot 0
    
    menu begin bt
    menu title BackTrack 5
    # ok here comes the real shit
    label backtrack5
    menu label BackTrack R1
    kernel bt5/vmlinuz
    append boot=casper netboot=nfs nfsroot=vulcano:/mnt/bt5 initrd=bt5/initrd.gz text splash vga=791 file=/cdrom/preseed/custom.seed --
    menu end
    

    you’ll need to copy to tftpboot/bt5 the initrd.gz and vmlinuz from the backtrack ISO /casper folder (which you can mount -o loop -t iso9660 bt5.iso /mnt/bt5

    the rest of the files you provide to the bootee over NFS

    /etc/exports:

    /mnt/bt5 10.10.3.0/24(rw,sync,no_subtree_check) 10.10.10.0/24(rw,sync,no_subtree_check)
    mount -t iso9660 -o loop BT5R1-GNOME-32.iso /mnt/bt5
    

    add a http server with kickstart / preseed files for an ever more powerful setup,
    in which case you replace the file= stanza in the append line with
    url=http://host/path/to/preseed

    more on preseeds… maybe later.

    Now restart all dependent services:

    /etc/init.d/nfs-kernel-server restart
    /etc/init.d/tftpd-hpa restart
    /etc/init.d/apache2 restart
    /etc/init.d/pxe restart
    

    debugging this setup usually requires tracing the process that is failing, so:
    - dhcp options tracing (dnsmasq verbose and tcpdump / wireshark)
    - verbose pxe
    - verbose foreground tftpd-hpa : in.tftpd -v -v -L /var/lib/tftpboot

    consolekit is evil

    Wednesday, December 1st, 2010

    … and hates me

    I should really tell you about the DLD seminar three weeks ago, or the PARANOIA security conference, or even just that Adobe should be considered harmful but things have been crazy and between this and electromagnetism I haven’t had the mind space. After the 6th of december, I promise I’l come back with pictures and relations and maybe even sounds (I have notes, don’t worry I’ll remember).

    On the other hand here’s a nasty hack to kill console-kit-daemon, which has a really nasty way of polluting the PID-space… and annoys me enough to warrant a public humiliation as well. What does it do, and why? Who cares what it does, it’s doing it poorly enough to catch attention to itself! So here’s how to kill it:

    root@wasp:/usr/sbin# dpkg -S console-kit-daemon
    consolekit: /usr/sbin/console-kit-daemon
    

    DON’T try to purge the package because that’s just one end of a really big ugly yarn of unneccessary dependency pain that I’d like to spare you…

    DON’T try to replace /usr/sbin/console-kit-daemon with your own stub… turns out dbus autostarts this “service”, and that approach will make dbus block your (ssh) session when you log in… not forever, but that’s even more annoying than the pid pollution.

    Instead, debian bug #544147 and #544483 clewed me in to the following hack:

    cp /usr/share/dbus-1/system-services/org.freedesktop.ConsoleKit.service \
      /usr/local/share/dbus-1/system-services/
    echo Exec=/bin/false >> /usr/local/share/dbus-1/system-services/org.freedesktop.ConsoleKit.service
    

    which is a two-liner, and would have been less ugly and easier to debug if it hadn’t been for the fine hubris of the freedesktop dudes…

    tune2fs and green disks

    Thursday, August 5th, 2010

    Hey folks,
    old news I’m sure, but if you get tempted into buying the new WD Caviar “Green Power” disks there is something you need to know about them: they fake 512-byte blocksizes while in reality having 4096-byte blocks! The move to 4K blocks is reasonable considering we just busted the 2 terabyte barrier, but the disk firmware is faking 512-byte blocks in the name of compatibility (read: so windows xp won’t shit itself).

    Unfortunately, running in bs512 mode makes the disk exactly 3x slower than it should be!
    The fix: line up your partitions at 4k boundries, so start partition one at block 64, 1024 or even 2048 (the win7 start block) not the default, 63, in most partitioning software. Start fdisk with the -u parameter and carefully specify the start block. In gparted you’ll have to unhook the “snap to cylinder boundries” checkbox, and then I suppose you could even move a partition to the right block, but expect this to take an inordinate amount of time!

    On a related note, fsck’ing an ext filesystem on boot is a drag, and fsck’ing 2TB file systems is a huge drag. Sure you should be running the fsck but it has a nasty tendency to happen on your workstation precisely when you can’t afford the extra 5 minute delay!

    I bump the default 10 mounts count to 0 (disabling mount count fscking) and auto-fsck my disks every 99 days, staggered so not all disks get checked on the same day. Do this with the tune2fs command:

    wasp:~# tune2fs -c 0 -i 99d /dev/sda1
    tune2fs 1.41.12 (17-May-2010)
    Setting maximal mount count to -1
    Setting interval between checks to 8553600 seconds
    

    out.

    PS I recently managed to achieve sustained throughputs of 110MB/s with these WD disks and properly aligned partitions:

    7516192768 bytes (7.5 GB) copied, 68.4392 s, 110 MB/s
    115+0 records in
    114+0 records out
    

    yes that’s disk-to-disk with ext4 and one large file, no fragmentation.

    PPS the defaults have nowadays changed to 120 days and 39 mounts, to which I say -1 mounts is better anyway!

    edit: Now that your files are aligned, you can specify a block size to mkfs as well, which might avoid unaligned fragments: mkfs.ext4 -b 4096 -L gigantor -O sparse_super /dev/sdb1

    the right way to use disk space? virtually, of course!

    Monday, December 14th, 2009

    I might have mentioned agedu before, a nice tool to find your least useful and ready-to-be-deleted files real quick.

    Sucks when the only files you have are rather large ones that you can’t throw out, like virtual system images which can easily become more than a few gigs heavy.

    Disk is cheap you say (again) and I will protest loudly; disk is not cheap for your laptop, it is not cheap for your high-performance platter server, it is not cheap for the environment and it’s ridiculous what kind of wasteful behavior the “hey, it’s cheap” mentality promotes, not all of which relates to computers (think garbage, cars, food, wars, lives…)

    Regardless, if you are using KVM there is a way to save disk space, speed up disk accesses and maybe even save the environment a little: kvm ships with a little tool called kvm-img (if you’re using QEMU then it’s qemu-img), and support for a copy-on-write storage format called QCOW2.

    The qcow2 format is cool because it supports compression and encryption.

    Compress your images

    If you cared about disk before, you could untick the “allocate all space now” and save a couple gigs on a 10G disk image, but that wouldn’t last long and you’d hear people grumble about disk corruption and such (corruption that I have never ever seen, I might interject), but now you can compress and rebase your image. Here’s how I saved 20G on my disk:

    To convert your raw image to qcow2 you would do:

    kvm-img convert -c -f raw -O qcow2 $IN ${IN%.img}_base.qcow2
    

    where $IN is your existing image and ${IN%.img}_base.qcow2 is going to be the name of your new qcow2 image. If you have NADA space left, convert into tmpfs (make sure tmpfs is mounted with sufficient size), remove the raw image and copy the new image out of tmpfs. That’ll free up some space.

    Rebasing

    But why stop there? I mentioned rebasing, and rebase we shall.
    The qcow2 format it is a little less cool for introducing really sucky snapshotting support, as applying and creating snapshots with kvm-img takes hours and is likely to fail! I don’t recommend trying kvm-img snapshot -c foo.qcow2
    However, the copy-on-write functionality of qcow2 lets us implement functional faux snapshotting with little effort.

    Copy-on-write means we can create an image sliver that only stores the changes from some read-only base image. Even better, we can layer these slivers! So, with the script I’ll introduce in a second, we can:

    1. Create or convert into a compressed base image. Name it foo_base.qcow2, eg “debian_squeeze_base.qcow2″. This is the master base, ideally made right after installing the operating system or whatevr.
    2. Create a usable sliver to store new data into: kvm-img create -b debian_squeeze_base.qcow2 squeeze_today.qcow2
    3. If you are using libvirt, update your /etc/libvirt/qemu/.xml disk source file to point to the ‘today’ image, and restart the libvirt daemon and virt-manager, to catch on to the changes
    4. To create a faux snapshot, just move the today image and rebase it like in step 2.
    5. To revert a faux snapshot, just replace today’s image with the snapshot.

    And here is my rebase script:

    kwy@amaeth:/var/lib/libvirt/images$ cat rebase_snap.sh 
    #!/bin/sh
    
    BASE=$1
    if [ ! -f $BASE ]
    then
       BASE=$1.qcow2
    fi
    if [ ! -f $BASE ]
    then
       echo "No base image $BASE"
       exit
    fi
    REBASE=${BASE%.qcow2}_`date +%F`.qcow2
    if [ -n "$2" ] 
    then
       REBASE="$2"
    fi
    mv $BASE $REBASE
    kvm-img create -f qcow2 -b $REBASE $BASE
    kvm-img info $BASE 
    kvm-img info $REBASE
    
    echo "$BASE -> $REBASE"
    

    Advantages

    • It takes 2 seconds to rebase and restore as opposed to 1 minute vmware snapshot or 4 hours to snapshot with qcow2
    • you don’t need fancy RAID or LVM tricks
    • You save space as opposed to shitty qcow2 snapshots and raw image copies
    • you can keep several versions or patchlevels of an operating system, and several application groups on the same operating system without having to reinstall the system – you already have a base image you can use!

    Caveats

    The experience should be pretty stable, but there is always room to shoot yourself in the foot. Here are a couple of ways you can make it hard for yourself:

    • don’t run out of disk space – it will corrupt your open images, regardless of format
    • don’t modify a base image that another image depends upon.
      Your base image knows nothing about its children (newer snapshots and ‘today’ images), so modifying the base image will cause all its children to corrupt into weirdness. That’s why the base image is “read only” and should be named appropriately.
    • don’t go down under the stairs!
    • don’t do stuff you don’t understand!
    • don’t tell me this ain’t new, cause I know!

    xtend your battery so y ou can GO ALL NITE

    Monday, September 14th, 2009

    K3ep going all n1te just like all that sp4m c0ming in through your mailbox.10 watts, it's a new record!

    10 watts, it's a new record!

    From joke to revolver as we say, I’ve noted that many of you find hacking away from power sources quite useful. Here’s how to keep at it longer with low power.

    (more…)

    HOWTO avoid pains with NetCom 3g USB on jaunty

    Sunday, September 6th, 2009

    Internet anywhere, ain’t it great?

    If you have one of those 3g netcome HSDPA USB dongles you might hve noticed how they don’t really work so well out of the box.

    After I had spent 4 hours trying to get the thing working Martin smugly told me these things should be plug’n'play and proceeded to… fail to get it working. oW hELL…

    Cutting to the chase and sparing you the gritty details I have a recipie for getting 3g working with the Netcom ZTE-MF636 USB dongle. This recipie should work in ubuntu jaunty and similar recent distros, and most of the instructions apply to other USB dongles too. Included are also all the tips you need to avoid spending 4 hours hammering your head against the wall…
    (more…)

    useful scripts for digging through code

    Monday, March 30th, 2009

    After a little bit of digging on my now seldom used desktop machine I found some useful scripts I’d share with you all. They’re real good if you’re in the CLI a lot trying to make sense of large amounts of code.

    Amongst others, a ~/.bashrc file to rule them all.

    The bash goodness:

    g foo: what lines of what files contain the word foo?
    cat somefile | gi foo

    .... : cd ../../..

    f foo: case-insensitive search for file foo.

    finn foo: find filenames containing ‘foo’. Sensitive but won’t print svn garbage.

    p foo: highlight processes matching ‘foo’

    fsym foo: search for symbol foo in all objects in subfolders.

    rsym foo: check if libfoo.so links properly

    Persistent vim windows.

    Further: gvimws allows you to always open new files in the same editor on the current desktop. It also closes the file in any other editor on any other desktop automatically.

    And the kicker: ve will allow you to quickly open the files you’ve been searching for (using gvimws)

    Putting it together

    kwy@amaeth ~$ dmesg | gi error
    1404:[27214.044072] iwlagn: Error sending REPLY_ADD_STA: time out after 500ms.
    kwy@amaeth ~$ cd /usr/src/linux
    kwy@amaeth /usr/src/linux$ f iwl-core
    ./drivers/net/wireless/iwlwifi/iwl-core.h
    ./drivers/net/wireless/iwlwifi/iwl-core.c
    # doubleclick-select midclick-paste the above line.
    kwy@amaeth /usr/src/linux$ cdd  ./drivers/net/wireless/iwlwifi/iwl-core.c
    kwy@amaeth  /usr/src/linux/drivers/net/wireless/iwlwifi$ g time out
    iwl3945-base.c:767:			IWL_ERROR("Error sending %s: time out after %dms.n",
    iwl3945-base.c:1590:			IWL_ERROR("Time out reading EEPROM[%d]", addr);
    iwl-eeprom.c:246:			IWL_ERROR("Time out reading EEPROM[%d]", addr);
    iwl-hcmd.c:186:			IWL_ERROR("Error sending %s: time out after %dms.n",
    #select-paste filename:lineno pair from above
    kwy@amaeth  /usr/src/linux/drivers/net/wireless/iwlwifi$ ve iwl-hcmd.c:186:
    # BAM you have found the problem

    The fsym and rsym functions are the kind of thing that you’ll never need unless you’re maintaining a large build with libraries that often break. fsym will find the object file that contains the symbol you’re looking for. rsym will do more than ldd and the compiler do to check if your lib is linking correctly.

    The all singing, all dancing vimrc file

    It’s swell. Switch buffers with F3/F4, do typeahead search and all kinds of crazy things you’ll never find out about until you happen to press the wrong button. Just like all of vim.

    cross-reference like CSI

    Friday, March 27th, 2009

    I was watching The Mentalist the other night and during the course of a murder investigation they ask the cute geeky cop to “cross-reference” the list of people with brown cars with the list of buyers of a specific toothpaste brand.

    She brings up a fancy display of two lists and punches the keyboard a few times, and suddenly a couple of entries in the list are highlighted – the murder suspects. So without further ado I bring you this ultimate crime-fighting tool:

    crossref
    (link updated)
    Usage: Download; mv crossref.pl.txt bin/crossref; chmod +x bin/crossref; crossref list1 list2

    If you’re on windows you’ll need ActivePerl or Strawberry Perl.

    It may not have any fancy graphics, but it’ll get the job done and hey – this isn’t a TV series.