Posts Tagged ‘linux’

PRADS 0.2.0 hits release

Monday, June 21st, 2010

PRADS – the Passive Realtime Asset Detection System has reached release with codename: “our two cents”.

It’s been far too long since last release and many things have happened that we thought we would share with you.
First off, PRADS has been rebuilt from scratch to handle high throughput and should work nicely on those fat pipes out there. This means it operates a little differently on the command line.
Our tool is now quite easy to use and has support for many more signature methods.

Changelog for prads 0.2.0-1
* PRADS release 0.2.0
* SYN,SYNACK,ACK,FIN,RST, IPv6, service, client, UDP, ICMP, ARP support
* added and fixed many signatures
* log to prads-asset.log
* eat pcaps (-r file.pcap)
* dump statistics on exit
* wirefuzz script
* prads2snort and other fun tools
* better IPv6 support
* better OS guessing
* awesome memory usage and stability
* l337 optimizations for high thruput
* code refactoring, cleanups & bugfixes and more

Quick start:
root@machine# prads -D
[*] Running prads 0.2.0
[*] Using libpcap version 1.1.1
[*] Using PCRE version 7.8 2008-09-05
[*] OS checks enabled: SYN SYNACK RST FIN ACK
[*] Service checks enabled: TCP-SERVER TCP-CLIENT UDP-SERVICES ARP
[*] Device: eth0
[*] Daemonizing...

To see the raw asset log file:

root@machine# tail -f /var/log/prads-asset.log
asset,vlan,port,proto,service,[service-info],distance,discovered
84.24.154.213,0,1268,6,ACK,[65392:118:1:0:.:A:Windows:XP],10,1277044697
109.87.38.106,0,56393,6,ACK,[16425:114:1:0:.:A:Windows:XP],14,1277044697
192.168.2.43,0,38359,6,SYN,[S4:64:1:60:M1460,S,T,N,W7:.:Linux:2.6 (newer, 7):link:ethernet/modem:uptime:2630hrs],0,1277044698
192.168.2.43,0,48065,6,ACK,[54:64:1:0:N,N,T:ZAT:Linux:2.6:uptime:2630hrs],0,1277044697
76.99.73.67,0,55834,6,ACK,[33069:48:1:0:N,N,T:AT:Linux:2.4(newer)/2.6:uptime:307hrs],16,1277044697
65.191.159.39,0,48747,6,ACK,[259:114:1:0:N,N,T:AT:unknown:unknown:uptime:20hrs],14,1277044697

Remember that ACK mode is and always will be rather unreliable.

To get a better view of the detected systems, run the following command:

prads-asset-report | less
13 ------------------------------------------------------
IP: 109.87.38.106
OS: Windows Server 2008 (R2 Standard 64-bit) (60%) 1
[..crop..]
104 -----------------------------------------------------
IP: 192.168.2.43
OS: Linux 2.6 (newer, 7) (100%) 3
MAC(s): 00:DE:AD:BE:EF:2F (2010/06/20 16:39:00)

Port Service TCP-Application
80 CLIENT Mozilla/5.0 (X11; U; Linux x86_64; en (US) AppleWebKit/533.4 (K
HTML, like Gecko) Chrome/5.0.375.70
80 CLIENT @www
80 CLIENT Mozilla/5.0 (X11; U; Linux x86_64; en (US) AppleWebKit/533.4 (K
HTML, like Gecko) Chrome/5.0.375.70
443 CLIENT TLS 1.0 Client Hello
443 CLIENT TLS 1.0 Client Hello
3218 CLIENT rtorrent/0.8.6/0.12.6
6667 CLIENT @irc
6667 CLIENT @irc
6667 CLIENT SSL 2.0 Client Hello
50005 SERVER Bittorrent
50005 SERVER Bittorrent

Port Service UDP-Application
53 CLIENT @domain
53 CLIENT @domain
123 CLIENT @ntp

105 ------------------------------------------------------

[..snip..]

Packages are available for debian and ubuntu, for everyone else there is source.
Get PRADS now!

Report issues and feature requests to: http://github.com/gamelinux/prads/issues

For suggestions, help, contributions and general banter go to the PRADS mailing list.

EDD DoS detection and DLD

Friday, April 16th, 2010

Hi all,
a short note about the Norwegian data surveillance directive that is up for passage into law these days. This directive, “Datalagringsdirektivet”, is the single most harmful threat to the general public’s privacy while being completely ineffective at stopping the bad guys it’s meant to target. Protests last Saturday in front of the parlimentary building – which yours truly attended – featured politicians and individuals from all ends of the political spectrum. Read more on Stopp DLD.

On to other things,

getting DDoSed sucks, as some of my collegues found out recently. Wouldn’t it be great if we could detect DDoSes as they come in through the wire? I mean besides when all of nagios goes code red upon us?

Well, I’ve written the little program that could. It’s not quite there yet (too few hours in the day) but the basic principles are fleshed out, and they go a little something like this:

There is a mathematical and a physical notion of entropy. To put it bluntly, it’s the shortest representation of a given piece of information. There’s a theorem that states that if you get many messages, but the messages put together don’t amount to much, then probably someone is fugging with you. We can use this to detect anomalies in network traffic, too.

This theorem about entropy is what EDD, the Entropy Distributed Denial of Service Detector [tarball] uses to classify a packet stream as bollocks, or not.

EDD is still pre-alfa software, which means that it’s a little too simplistic to tell you anything beyond a mere “Something’s up”, but I’d like you to test it in your setups with the understanding that the program.

commit bc2f4df34745e4c422a17e70aac271bc930b9f1a
Author: Kacper Wysocki
Date: Fri Apr 16 18:18:37 2010 +0200

EDD now classifies simple SYN floods successfully.

* faster and simpler simple_entropy
* reads from pcaps (-r)
* configurable treshold (-e)
* configurable window size (-w)
* profile counting (-t)
* edd self-tests (-E)
* better TODO ideas

Try it out and let me now, and send me pcaps of your DDoS and false positives.

bifrost virtual appliance

Monday, April 12th, 2010

I’ve just released a Virtual Appliance for Bifrost 1.2 to ease deployment of Bifrost, the single-print-queue system.

This should make it easier to give Bifrost a go in your organization. Read more on the Bifrost Virtual Appliance page.

brilliant fools – hackers update

Monday, March 1st, 2010

cracks are on the rise. so are hacks, and I haven’t posted a thing since December. So what’s up with you?

Good news is that Con Kolivas might have managed to defeat his carpal tunnel and swallow his spite for kernel dev elitism, and is again churning out solid kernel code to improve desktop usability – which the kernel devs aren’t too interested in – something he is quite right to say!

Hopefully ubuntu will pick up CK’s scheduling patches, because they are uber and with ubuntu’s momentum they might topple the stack. Too bad their kernel team can’t follow the churn. Wish I had time to compile it for you or even post some interbench stats with pretty graphs, but there is a BFQ PPA available already so you can test it out on your ubuntu or debian machine. Bonus? Some random dudes wrote a simple IO scheduler which is included there. Ain’t that reasonable?

All that at least until we have time to write our own OS which gets rid of all suckage, is super flexible and of course incorporates all our favorite patches.

Valgrind is pretty neat, and with that we have working stray acks in PRADS, the stealthy host and service detection system. More features and even a new release might come soon, which is to say – when it’s ready. In the mean time I welcome you to try breaking it in any and every way possible.

Also, if everything you do is motivated by monetary gains then you sir are a shame to the human race. Go back to step 1 and have a good day now.

the right way to use disk space? virtually, of course!

Monday, December 14th, 2009

I might have mentioned agedu before, a nice tool to find your least useful and ready-to-be-deleted files real quick.

Sucks when the only files you have are rather large ones that you can’t throw out, like virtual system images which can easily become more than a few gigs heavy.

Disk is cheap you say (again) and I will protest loudly; disk is not cheap for your laptop, it is not cheap for your high-performance platter server, it is not cheap for the environment and it’s ridiculous what kind of wasteful behavior the “hey, it’s cheap” mentality promotes, not all of which relates to computers (think garbage, cars, food, wars, lives…)

Regardless, if you are using KVM there is a way to save disk space, speed up disk accesses and maybe even save the environment a little: kvm ships with a little tool called kvm-img (if you’re using QEMU then it’s qemu-img), and support for a copy-on-write storage format called QCOW2.

The qcow2 format is cool because it supports compression and encryption.

Compress your images

If you cared about disk before, you could untick the “allocate all space now” and save a couple gigs on a 10G disk image, but that wouldn’t last long and you’d hear people grumble about disk corruption and such (corruption that I have never ever seen, I might interject), but now you can compress and rebase your image. Here’s how I saved 20G on my disk:

To convert your raw image to qcow2 you would do:

kvm-img convert -c -f raw -O qcow2 $IN ${IN%.img}_base.qcow2

where $IN is your existing image and ${IN%.img}_base.qcow2 is going to be the name of your new qcow2 image. If you have NADA space left, convert into tmpfs (make sure tmpfs is mounted with sufficient size), remove the raw image and copy the new image out of tmpfs. That’ll free up some space.

Rebasing

But why stop there? I mentioned rebasing, and rebase we shall.
The qcow2 format it is a little less cool for introducing really sucky snapshotting support, as applying and creating snapshots with kvm-img takes hours and is likely to fail! I don’t recommend trying kvm-img snapshot -c foo.qcow2
However, the copy-on-write functionality of qcow2 lets us implement functional faux snapshotting with little effort.

Copy-on-write means we can create an image sliver that only stores the changes from some read-only base image. Even better, we can layer these slivers! So, with the script I’ll introduce in a second, we can:

  1. Create or convert into a compressed base image. Name it foo_base.qcow2, eg “debian_squeeze_base.qcow2″. This is the master base, ideally made right after installing the operating system or whatevr.
  2. Create a usable sliver to store new data into: kvm-img create -b debian_squeeze_base.qcow2 squeeze_today.qcow2
  3. If you are using libvirt, update your /etc/libvirt/qemu/.xml disk source file to point to the ‘today’ image, and restart the libvirt daemon and virt-manager, to catch on to the changes
  4. To create a faux snapshot, just move the today image and rebase it like in step 2.
  5. To revert a faux snapshot, just replace today’s image with the snapshot.

And here is my rebase script:

kwy@amaeth:/var/lib/libvirt/images$ cat rebase_snap.sh 
#!/bin/sh

BASE=$1
if [ ! -f $BASE ]
then
   BASE=$1.qcow2
fi
if [ ! -f $BASE ]
then
   echo "No base image $BASE"
   exit
fi
REBASE=${BASE%.qcow2}_`date +%F`.qcow2
if [ -n "$2" ] 
then
   REBASE="$2"
fi
mv $BASE $REBASE
kvm-img create -f qcow2 -b $REBASE $BASE
kvm-img info $BASE 
kvm-img info $REBASE

echo "$BASE -> $REBASE"

Advantages

  • It takes 2 seconds to rebase and restore as opposed to 1 minute vmware snapshot or 4 hours to snapshot with qcow2
  • you don’t need fancy RAID or LVM tricks
  • You save space as opposed to shitty qcow2 snapshots and raw image copies
  • you can keep several versions or patchlevels of an operating system, and several application groups on the same operating system without having to reinstall the system – you already have a base image you can use!

Caveats

The experience should be pretty stable, but there is always room to shoot yourself in the foot. Here are a couple of ways you can make it hard for yourself:

  • don’t run out of disk space – it will corrupt your open images, regardless of format
  • don’t modify a base image that another image depends upon.
    Your base image knows nothing about its children (newer snapshots and ‘today’ images), so modifying the base image will cause all its children to corrupt into weirdness. That’s why the base image is “read only” and should be named appropriately.
  • don’t go down under the stairs!
  • don’t do stuff you don’t understand!
  • don’t tell me this ain’t new, cause I know!

kernel coolness, finally!

Thursday, November 5th, 2009

Many things worth blogging about are happening lately! In fact, so many things that there is not enough time to blog about them. Ah, where to begin!

Quickly now:

PRADS

Ebf0 and myself had a lecture about our fine host detection application at Dagen@IFI (Institute for informatics, UiO). Presentation available here, at least until we upload it to the project website.

We now know that our Proof of Concept is k00l and Ebf has started the high-performance C implementation.

Kernel hacks

Did you know you’re missing out on cool kernel features? Well, yes you are. Here are some of them:

  • grsecurity : Better security in linux! Fixes thousands of attack vectors for desktops and servers alike
  • compcache: compressed memory swap might sound counter intuitive, but memory is lightning fast compared to disk, and you can cram more apps into compressed memory!
  • nilfs: Every wished you hadn’t deleted that file 5 seconds ago? Or wasted an hour waiting for a fsck? Log structured file systems scream write performance. And NILFS aids in data recovery too, as it’ll take automatic snapshots of your data every synchronous write. Very sweet.
  • reiser4: Don’t get me started. This is still not reached mainline. Hans be damned. However, reiser4 is still the fastest file system around.
  • ++++ low-latency, preemptible, tickless system, loads of hardware support and lots more!

The upshot?

The -lied patchset is back!

I now track Ubuntu karmic git and I provide i686 packages:

Add the following to your /etc/apt/sources.list :

deb http://www.cs.mcgill.ca/~kwysoc/debian/binary ./

then install the package:

# add the archive key:

gpg --recv-key 089ac586 && gpg --armor --export 089ac586 | sudo apt-key add -

# update package database:

sudo apt-get update

# install the kernel package

sudo apt-get install linux-image-2.6.31.3-lied-grsec-dirty

# check grub or lilo and then reboot into the kernel!

amd64 binaries are coming as soon as I get a chance to compile them. For now grab the karmic git, the patchset and .config and roll your own :-)

What else?

Bifrost is coming along, and might be close to a release soon, and

Multiframe needs a new client release (which I am w0rking on)

oh and I’ve made an

auto-migrate from ISC dhcpd to dnsmasq by script

…with my quick and dirty perl f00 : dhcpd2dnsmasq.pl.

Be mindful that it is best for those with a lot of host definitions, and does not support all the ISC syntax out there.

The script is interesting because it consicely illustrates how to make a simple but powerful parser with the minimal amount of lines (and fuss) using the AND-OR Waterfall method.

Honk and Drop me a comment if you like / hate / fake it~!

xtend your battery so y ou can GO ALL NITE

Monday, September 14th, 2009

K3ep going all n1te just like all that sp4m c0ming in through your mailbox.10 watts, it's a new record!

10 watts, it's a new record!

From joke to revolver as we say, I’ve noted that many of you find hacking away from power sources quite useful. Here’s how to keep at it longer with low power.

(more…)

HOWTO avoid pains with NetCom 3g USB on jaunty

Sunday, September 6th, 2009

Internet anywhere, ain’t it great?

If you have one of those 3g netcome HSDPA USB dongles you might hve noticed how they don’t really work so well out of the box.

After I had spent 4 hours trying to get the thing working Martin smugly told me these things should be plug’n'play and proceeded to… fail to get it working. oW hELL…

Cutting to the chase and sparing you the gritty details I have a recipie for getting 3g working with the Netcom ZTE-MF636 USB dongle. This recipie should work in ubuntu jaunty and similar recent distros, and most of the instructions apply to other USB dongles too. Included are also all the tips you need to avoid spending 4 hours hammering your head against the wall…
(more…)

never ask for root again

Monday, August 17th, 2009

just a short note to all of you:

linux is not secure. Passwordless root is here :-*

Yes, it has been published elsewhere, but I’ll do mine to push this meme to you: there can be no “untrusted local users” nor do I believe that your services aren’t exploitable.

Two seconds later I have root on your box.

Despite LSM. Despite SELinux. Despite jails and virtualization. Despite all your assumptions.

You will need some very fine security gents and a little of your own smarts to secure your nets. Call us :-)

The best link on this issue so far has been:

cr0: bypassing linux with null pointer

Do you want security? Go run carpal-tunnel-inducing OpenBSD, swell swell if only it smelled well FreeBSD, or, *drum rolls*

drop-in up-to-date secure and invulnerable grsec kernel for ubuntu and debian

Only disadvantage I can see is that they don’t provide amd64 and desktop builds.

Dilligence and perseverence is the path to victory,
and although paranoia may not be the path to safety
noone should leave their front door open.

In other news, and probably a little lame for those of you coming thru the planet feed, security.vcl is here – properly used, understood and abused it could save you some worries, making sure no “untrusted user” went “local” in the first place.

Also, tell your friends: there is a Facebook virus about. It sends links to you from your friends accounts. If you click on the link, you too will be sending your friends links.

Yeah, I know, that sounds like what I do on facebook all day. Except the difference is you don’t know you’re sending links.

So watch out.

And tell your less savvy friends.

how to break your head : try linux compilination

Wednesday, July 29th, 2009

I’ve recently started compiling my own kernels again. Some people ask me why I’d ever want to do this – a valid question, since anyone who’s done it knows a time-consuming hassle best left to the distro packagers and really nerdy people with too much time on their hands. Other people will give a blank face and ask “What is a Conpiling?” To these other people: this article is not for you, it will only serve to confuse that pretty little head of yours. If you know what ‘a compiling’ is, you may proceed. I don’t provide references; I banter them. Google your friend, pluckum.
\
Still, I am not here to discuss the reasons for compiling your own kernel – these are all too obvious to the initiated and completely uninteresting to anyone else. I’m more interested in the reasons why my friends, collegues and I have *stopped* compiling our own kernels – despite some of us enjoying at least a compile a day (or ten!) for periods of time in the past. Only the gentoo rice boys remain, steadfastly compiling everything in sight despite snide comments about mean time between upgrades and ridicule about their USE_FLAGS selector GUIs.
\
Why don’t we compile anymore?
There is no stable upstream branch. In my own experience this has had direct consequences for the stability and quality of point releases.
Years after Linus’ bitkeeper schism, the SCO slimeballing and the death of the stable branch, we can look back and say that aye, we have a better audit trail and development has scaled through the roof. We have more kernel features than ever, and an astounding rate of patches make it into mainline every day.
\
These amazing developments are a long shot away from the linux dev process back in the days of 2.2 and 2.4, but there is a dark side to these developments.
Regressions are no longer the domain of the bleeding edge, the -mm or -ac trees, -alpha and -rc releases for the adventurous, masochistic or desperate. Common things. Getting bitten by that local sexploit and being too embarassed to tell your friends about it. Software suspend used to work fine. The graphics card did not crap itself on the last point release, but at least my NIC doesn’t get bricked in this one. The wifi keeps screwing with you, but you don’t know if you should blame Ubuntu, Intel or Linus. On the internet noone can hear you scream.

\

Elitism is rife on the LKML, and more pointedly, in the mainline patch process. Who knew NIH would be such a big problem in an open source project? Admittedly, it is the largest and perhaps the most ambitious open source project of all, with all eyes on target, a million uses and powerful market forces pulling the project this way and that. Linux has long ago outgrown the boy’s room, the hacker dungeon and its academic roots. Most kernel patches that get into mainline are pushed there by large hardware and software vendors. Many kernel hackers hack the kernel on their day job, earning an engineer’s living.
\
Linux has reached the Enterprise in a big way. The system runs and is optimized for Big Iron. The desktop is “good enough”, say the kernel hackers. Latency is fine for our uses, and those squeaky audiophiles should shut up and fork. Indeed they did, as embedded, realtime and audio people have all collectively decided to jump off the wagon.
Out-of-tree kernel hackers already know where the lay is at. After years of pushing the same genious useful patchsets they are sick of cleaning up, splitting out, documenting, backporting, forward porting only to discover that noone read their patch. Maybe they will be lucky, their ideas bastardized overnight into someone else’s pet project, far more likely to succeed once it is Invented Here(tm).
\
It’s not all bad: we want and need to trust the people that push stuff into the kernel. Who are you to think that you can do it better than them? They are doing their job, they do it well, so what if they all meet for beer and virgin sacrifice after hours, so what if there is no free seating in their society? Fork your own.
\
Weiging in at 800MB uncompressed, the Linux source is a behemoth. Counting only source, headers and assembly, there are 35,000 files in the linux kernel, with 10,667,648 lines of source code. This code is metriculously organized, not only into systems, subsystems and modules, but into domains of responsibility. Hey, if you’ve ever managed a large software project you would know how annoying, how encroaching it is when someone start fiddling with your private bits.
\
On the other hand, linux has lost a lot of great contributions and spurned a lot of marvelous people because of this elitism. OpenMosix israeli clustering, reiser4 the murderous file system, software suspend 2 the ‘it just works’ approach, page-in-from-swap, CK’s desktop efforts, the two kernel monty carlo and process snapshotting are only few of the projects that failed to sufficiently influence the core developers, some of them even year after year.
It can be argued that despite the patches not making it to mainline some of these ideas did find their way into the minds of the gitmasters and found other implementations on technical merit alone. To me this defeats the whole purpose of the open source model which drives technology by sheer speed. We’ve had a working, cleaned up, documented version of the patch for two years – and the feature doesn’t make the cut. This is too little too late.

\

Well, not everyone takes an interest in kernel politicking even if they follow the LKML or kerneltrap, and some people even like hitting bugs and fixing issues in their compiles, and trolling in epic flame wars. They too have left kernel compiling to other, more patient and masochistic people.
Maybe it’s because even grepping a single point release changelog is a major chore. The distro folks have gotten fairly good at kernel compiles; ubuntu ships a one-size-fits-all Just Works(tm) kernel, RedHat’s patchset has grown less offensive over the years and debian is and always was debian. Upgrades are relatively painless and usually somebody else already did the dirty work.
Linus Thorvald’s initial plan succeeded: by axing the stable/unstable tree he told the world that the responsibility for stability rests on the distributor. He also axed many hobbyists’ will to stay and play with new releases. I’d rather go play on milw0rm.

\

There are other compelling reasons not to roll one’s own: the number of configuration options has doubled over the past years, and most of these new options are not relevant to the hobbyist use case. Development not only in the kernel source but in the toolchain (gcc) has caused compile times to soar. I remember proudly compiling 2.4 kernels on my K7 within 10 minutes back in 2001. Today it might take longer to compile the tree on my Centrino dual-core.
And there it is: we’ve suffered feature creep and bloat. After a long download, an hour or more of configuring, and many failed initial make runs, a generic compiled bzImage weighs in at about 3412 kB. This is a modular kernel, mind you. What happened to lean and mean 800 kB kernels?

Memory is cheap you say.
But minds are not cheap!

\

I’m announcing a contest: what’s the smallest stable useful kernel you can make for your platform? Remember, it should run on other machines and be useful, and the compile reproducible. Choose your own definition of useful, but do find a concrete definition. Use any tree and patchsets that turn you on. Bonus points for packaging so others can plug your kernel into their system. I’ll make your package available.
As a side contest I’ll take compile times along with bogomips numbers and your .config file for reference.

\\\

PS. Yahoo! internal IT sucks. Where’s the wifi? Running our own cables, canned XP images in a linux lab, packet loss. This aint no funky party. I guess they are too busy. Paranoia maybe. Things aren’t wonderful.