Archive for the ‘security’ Category

Full-stack Varnish all the things

Friday, December 25th, 2015

While attending VUGX Rotterdam, which was a jolly time for the Varnish family, I have been pleasantly surprised by some little known but very useful new things available for your site.

Varnish Security Firewall

I’ll tell you about them in a moment, but I’ll start by plugging VSF: The Varnish Security Firewall, which is a really useful security framework for Varnish 4, which we recently made really, really easy to install. You can give it a spin right now to stop many of the most common attacks against your website today.

Shameless plugs out of the way, Varnish is a painless process to operate, but most sites out there need to admin web servers, they need to encrypt their traffic and they need to load balance, and all this together becomes pretty messy, requiring haproxy, nginx and sticky putty goo in a glorious sandwich.

Nginx Sandwich

SSL frontend with Hitch

One problem that keeps popping up is the lack of SSL support. No, Varnish is not going to link openssl before hell freezes over in PHK’s basement, because let’s be honest: it belongs in its own binary. Still, you can now replace your heavy-handed Nginx stack with the lightweight Hitch, a revamped stud, which even has rudimentary Let’sEncrypt support.


An SSL frontend config for Hitch is a simple 5 lines.

frontend = "[*]:443"
backend  = "[]:8000"
pem-file = "/etc/ssl/private/projects.hackeriet.pem"
user = nobody
write-proxy = on

With the write-proxy = on option, we can even use the PROXY protocol to pass the client.ip to Varnish instead of trusting X-Forwarded-For, but we’ll have to update Varnish to accept it by copying /lib/systemd/system/varnish.service to /etc/systemd/system/varnish.service and editing ExecStart to include -a:8000,PROXY like so:

ExecStart=/usr/sbin/varnishd -a :80 -a:8000,PROXY -a :6081 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m

On the backend side, it’s fairly straightforward to write SSL-backends for Varnish using stunnel.

Varnish as a Web Server

For years we’ve been half-joking that what Varnish needs is web server capabilities, but little did we know that this has been around for years!

web server

Today we can all enjoy Martin’s fsbackend vmod, which lets us serve (and cache) static files straight off the file system in a secure and fast way. It doesn’t come with much documentation though, so here’s how you use it to serve your sites with the standard “index.html”, some minimal trailing-slashes support and some content-typing:

vcl 4.0;
import fsbackend;
sub vcl_init {
  new webroot = fsbackend.root("/var/www");
  webroot.add_header("Cache-Control: public, max-age=3600");
sub vcl_backend_fetch {
  if(bereq.url ~ "^/mysite") {
    set bereq.url = regsub(bereq.url, "/mysite/?", "/");
    if(bereq.url ~ "/$") {
      set bereq.url = bereq.url + "index.html";
    set bereq.backend = webroot.backend();
sub vcl_backend_response {
  if (bereq.url ~ "\.html") {
    set beresp.http.Content-Type = "text/html";
  } elif(bereq.url ~ "\.txt") {
    set beresp.http.Content-type = "text/plain; charset=utf8";
  } else {
    set beresp.http.Content-Type = "text/plain";


This module is historically proceeded by Dridi’s efforts for Varnish 3 and 4, which by his own words was “a hack”, as well as the std.file(), which was not good enough for serving web pages.

Coupled with Hitch for SSL, this has allowed me to drop the Nginx sandwich completely on the static parts of my sites.

Regexes with sub-captures


Innate regexes are awesome in Varnish, but sub-captures are barely accessible and usually force you to copy and match a single string many times for multiple captures. This is both messy and not efficient. Enter Geoff from Uplex re vmod, which lets us access all subcaptures of a efficiently:

import re;
sub vcl_init {
  new myregex = re.regex("bar(\d+)");
sub vcl_recv {
  if (myregex.match(req.http.Foo)) {
    set req.http.Baz = myregex.backref(1, "");

Consistent Hashing

If you’re running a really large site where you’re regularily pinning your server bandwidth you know that load-balancing and sharding is where it’s at. That’s where the stateless persistent hashing vmod that Niels from Uplex presented comes in.
Consistent Hashing
When a frontend request hits your cluster, the vslp vmod looks up which cache is the master for that request, and then picks which backend owns the request in a consistent way across n servers. If you ever add frontends or backends, only 1/n+1 objects have to be rebalanced, giving you a limited increased load cost for adding new servers. The code has been in heavy production for the last couple of years!

This means you can keep adding frontends and backends to your pool, and you can round-robin load balance while scaling (near-) linearly.

Dynamic Backends

When running Varnish on EC2 with an ELB, it’s real tricky to keep the backend list up to date. Dynamic backend vmods will fix this problem Real Soon Now.

With consistent hashing in the Varnish core, and dynamic backends, we might no longer need haproxy nor any other messy load balancer, simplifying the stack for quite a lot of sites out there.

fullstack of pancakes
There are also rumours of a FCGI or even WSGI vmod, which will allow us to forgo the ancient dogmatic cows of web adminning entirely, and go Full Stack Varnish. Let’s just pray that day will come soon enough.

Ho Ho Ho! I wish you a merry X-Forwarded For!

PS The title is a joke, and so is everything “full stack”.

NSA-proof SSH

Tuesday, January 6th, 2015

ssh-pictureOne of the biggest takeaways from 31C3 and the most recent Snowden-leaked NSA documents is that a lot of SSH stuff is .. broken.

I’m not surprised, but then again I never am when it comes to this paranoia stuff. However, I do run a ton of SSH in production and know a lot of people that do. Are we all fucked? Well, almost, but not really.

Unfortunately most of what Stribika writes about the “Secure Secure Shell” doesn’t work for old production versions of SSH. The cliff notes for us real-world people, who will realistically be running SSH 5.9p1 for years is hidden in the repo.

Edit your /etc/ssh/sshd_config:

Ciphers aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-512,hmac-sha2-256,hmac-ripemd160
KexAlgorithms diffie-hellman-group-exchange-sha256

Basically the nice and forward secure aes-*-gcm chacha20-poly1305 ciphers, the curve25519-sha256 Kex algorithm and Encrypt-Then-MAC message authentication modes are not available to those of us stuck in the early 2000s. That’s right, provably NSA-proof stuff not supported. Upgrading at this point makes sense.

Still, we can harden SSH, so go into /etc/ssh/moduli and delete all the moduli that have 5th column < 2048, and disable ECDSA host keys:

cd /etc/ssh
mkdir -p broken
mv moduli ssh_host_dsa_key* ssh_host_ecdsa_key* ssh_host_key* broken
awk '{ if ($5 > 2048){ print } }' broken/moduli > moduli
# create broken links to force SSH not to regenerate broken keys
ln -s ssh_host_ecdsa_key ssh_host_ecdsa_key
ln -s ssh_host_dsa_key ssh_host_dsa_key
ln -s ssh_host_key ssh_host_key

Your clients, which hopefully have more recent versions of SSH, could have the following settings in /etc/ssh/ssh_config or .ssh/config:

Host all-old-servers


Note: Sadly, the -ctr ciphers do not provide forward security and hmac-ripemd160 isn’t the strongest MAC. But if you disable these, there are plenty of places you won’t be able to connect to. Upgrade your servers to get rid of these poor auth methods!

Handily, I have made a little script to do all this and more, which you can find in my Gone distribution.

There, done.

sshh obama

Updated Jan 6th to highlight the problems of not upgrading SSH.
Updated Jan 22nd to note CTR mode isn’t any worse.
Go learn about COMSEC if you didn’t get trolled by the title.

Grsecurity on the desktop

Friday, August 8th, 2014

In my last post I presented Grsecurity kernel packages for Debian Wheezy. Now would be a good time to review the hows and whys of Grsecurity, so you can decide if it is something you need.

Today we will quickly look at Grsecurity’s viability and impact on a typical desktop or laptop.

Why do I need Grsecurity on the desktop?

  • Often run unsecure code? Limit its impact on the system.
  • Employ chroots and containers? Enforce stricter containment.
  • Connect to hostile networks? Reduce and mitigate impact of exploitation attacks.
  • Allow others to use your system? Increase monitoring and control over your machine.

Or perhaps you choose to run Grsecurity on your laptop simply for the sheer paranoia-factor and to impress friends.

How does Grsecurity behave on the desktop?

In addition to the invisible yet significant hardening efforts againt kernel exploitation, there are some changes that an experienced user will notice immediately:

  • you need root to dmesg
  • grsec reports denied resource oversteps in dmesg
  • top only shows this user’s running processes
  • mappings and other sensitive information is only available to the process owner in /proc

The very few programs that depend on a specific kernel version, or that read /sys/kcore or write directly to kernel constructs will not work under Grsecurity.

And of course, there is the feeling of solidity and the sight of the reported kernel version:

~# uname -a
Linux amaeth 3.2.60-grsec-lied #1 SMP Wed Aug 6 17:40:27 CEST 2014 x86_64 GNU/Linux

kernel security for debian

Wednesday, August 6th, 2014

TL;DR – links to Grsecurity-enabled up-to-date debian wheezy-kernel packages at the bottom of this post.

Kernel security is becoming more important nowadays, especially since the Linux kernel has grown so big and the platform is rife with misbehaving programs.

Some enthusiasts have illusions that the Linux kernel is somehow magically very secure, but those in the know will tell you there are quicker ways to gain root than sudo.
you did what?
Grsecurity is by far the best patchset for increasing the security of a system, whether it be a virtual LAMP box, a login shell server or a desktop machine. I have been tracking it and using it for years and consider it superior to the SELinux and AppArmor approaches.

It has only one major drawback: most distributions of Linux do not provide tested and up-to-date grsec-patched kernel packages, making Grsec-level security features nearly unobtainium for the common mortal.

I have been rolling my own kernel patches since the millenium and so I put in the work to put Grsecurity back into Debian.

So far I have built and tested kernels for Debian 7.5 and 7.6 Stable codenamed Wheezy. This is the standard, debian-patched kernel with added Grsecurity.

I have built separate packages which are suitable for servers, virtualized servers and desktops, and these have been tested on run-of-the-mill LAMP boxen as well as custom and well-trafficed shell servers, and of course my trusty desktops and laptops.

Download and Install

You can download and install the grsec desktop. server and virtual server debian packages today!

Note, to avoid running out of space in /boot, change MODULES=most to MODULES=dep in /etc/initramfs-tools/initramfs.conf

Install the lied apt repository

sudo -i
wget -O- | apt-key add -
wget -O /etc/apt/sources.list.d/lied.list
apt-get update

Grsecurity Desktop/Laptop HOWTO

apt-get install linux-image-3.2.60-grsec-lied

Grsecurity Server HOWTO

dpkg -i linux-image-3.2.54-grsec-nose-server_3.2.54-grsec-nose-server-1_amd64.deb

Grsecurity Virtual Server HOWTO

dpkg -i linux-image-3.2.54-grsec-nose-virtual_3.2.54-grsec-nose-virtual-1_amd64.deb

Furthermore I commit to also merging the patchsets and making available Grsecurity packages for Debian 8/Jessie and providing it all in a debian repo. I will then make this available in a repo so that people can easily add it to their setup.
I also commit to keeping these packages up to date on all the platforms I care about.

Quick Howto build your own
To build your own Grsec-enabled debian kernel packages, execute the following commands:

gpg grsec-201408032014-debian-3.2.60-1.patch.gpg
apt-get source linux
cd linux-3.2.60
patch -p1 < ../grsec-201408032014-debian-3.2.60-1.patch
wget -O .config
make deb-pkg LOCALVERSION=-myversion

You can replace “grsec-config-server” with “grsec-config-desktop” or “-virtual” if you need one of the other configurations.

prads-0.3.2: ya skipped that one

Monday, November 5th, 2012

Ever since HACK.LU (where we spoke about VSF), Ebf0 and I have had quite some activity on PRADS, wonder why?

We really enjoyed the design of POM-NG, we find this little program quite inspiring and will keep in touch with GMsoft.

This might be the right time to announce PRADS-me! at, a service to actively fingerprint your own self. Real useful even just for an IP check, geolocation or to see what types of fingerprints you are matching at any given time.

Some of you might recall that PRADS was the subject of a Masters thesis in 2011: “Investigating Passive Operating System Detection” by Petter Bjerke Falch from UiO. Well, it’s happened again.

Jostein Haukeli at the University of Oslo Department of Informatics has written a paper on “False positive reduction through IDS network awareness”. We are excited about the prospect that our work is being used in data correlation, and we would like to see more event correlation stuff done in a scalable context.

Last year PRADS was a featured ip6-ready tool at the ISC.
Furthermore, in July this year PRADS was included in OSSIM, the Open Source SIEM

In other news, PRADS is about to be baked into the next release of the Security Onion network monitoring linux distro. Version 12.04 beta already comes with PRADS included (replacing old-timers sancp and pads) but it did require some bug-squashing from our end. You know what that means? 0.3.2-rc1 was tagged in the tree recently. That’s right: a new PRADS release is coming up Real Soon Now.

the paranoid console viewer

Tuesday, June 19th, 2012

Hi all,
I know it’s been a while since my last post.
There is lots to talk about, but let’s start afresh with something short and sweet.

Let me paint a picture for you:
There is something broken in the most holy and secure part of your network. You need to call support to have them look at it. The support rep wants console access, but you can’t give them axx to your holiest cream pie.
They offer to take over your desktop with a java rootkit app like TeamViewer, GoToMeeting or WebEx.
You decline. You need to stay in control, but show them what they need to see, that and only that.

Let me be clear on the problem statement:
Read-only shell access to the most secure host, which is not available over the wire, viewed by multiple parties at the same time.

Here’s how to do that with SSH, screen(1) and some foo,
with ssh->chroot->rbash->readonly multiuser screen->reverse ssh->openvpn:

You will need a linux server in an “unsafe” zone which is exposed to your support rep on the internet or thru VPN.

  1. Create the user to be contained on your unsafe box, with the restricted bash shell:
    unsafe# export user=rep; adduser $user; chage -s /usr/bin/rbash $user
  2. (Bonus:) chroot/contain the user within sshd_config
  3. Setup multiuser screen on the unsafe box. There are lots of guides for it, but the short and sweet of it is: unsafe# chmod +s `which screen`; chmod 755 /var/run/screen Indeed, this increases the attack surface, and therefore we call this box the unsafe one.
  4. ssh from secure zone to unsafe server:
    secure# ssh -R 2222:localhost:22 screen
  5. Run screen from YOUR account and do :addacl $user :chacl $user -w "#" :chacl $user -x "?" Replace $user with whatever from step 1. Then, still in your screen: :multiuser on
  6. Win! Now you can reverse ssh back to the secure zone and let $user on the unsafe box read the terminal without being able to access anything but what you show her.
  7. Bonus: Add `screen -r $youraccount` in $user/.profile and $user will drop straight into locked screen, and remember that multiuser screen is read-write-execute for all accounts that are addacl’d
    so you might want to chacl before enabling the $user account login.

    And there you have it, superparanoid reverese secure-unsecure remote shell viewer.


CPM 0.26 the Console Password Manager

Monday, December 5th, 2011

Some of you might have noticed that I’ve adopted this little program while its original author is MIA, and that my efforts have resulted in its inclusion into debian wheezy earlier this year.

This is great news and makes it a breeze to get up and running with CPM with a simple apt-get install cpm

However, it seems that most people are interested in running CPM on older distributions, realistically the stable distribution codenamed squeeze is a favorite, as well as the Ubuntu LTS release 10.4 codenamed lucid lynx.

So I have built some updated packages of CPM for these oldies but goodies:
* CPM for squeeze i386
* CPM for squeeze amd64
* CPM for lucid i386
* CPM for lucid amd64

Remember to install the dependencies though. On squeeze, they are:

me@mine:~# apt-get install \
    libcdk5 libcrack2 libdotconf1.0 libgpg-error0 \
    libgpgme11 libxml2 libxml2-utils libpth20

File us a ticket if you run into trouble with these packages or need cpm working on some other distribution.

CPM is a simple, paranoid password manager for the console with some cool features that make it stand out:

* data files can be encrypted for more than one person
* data files are signed by the last person who saved it so forging data files is not possible
* data files are en- and decryptable directly by gpg and gzip
* the application memory is protected from paging, core dumps, ptrace attacks and runtime environment
* data is validated using an internal DTD
* several passwords per account are possible to store
* it’s possible to handle several data files, each encrypted for different people
* cracklib checks of password strength and warnings about weak passwords
* user definable hierarchy with unlimited depth
* long comments for any node in the hierarchy
* password generator
* only one password visible at a time
* searchable database from the command line
* user definable search patterns (e.g. user@hostname)
* several hits can be displayed at once (e.g. several accounts per host)
* conversion scripts for Password Management System (pms), Password Safe and CSV files

oh noes, o cert my *sniff* cert

Monday, September 26th, 2011

papieren bitteI’m not going to tell you about DigiNotar, whose file of bankruptcy this month held shock for no one after recently having lost the keys to the grand vault, in which the government held much stock. Though I have many comments upon the sophistication of the player that so thoroughly owned the most trusted agencies of the digital age….

The cracker hardly needed them skillz, considering it has been a challenge to keep that whole corrupt industry accountable. The trouble with the central authority system is that even if only one of the keys is compromised, the system is broken and gives no assurances whatsoever. No warning bells either. Just a sweet silent man in the middle, passing along all the best parts to his lover.

It’s not a joke for the 300,000+ people who documentedly had their emails and facepalms compromised. We thought he was kind to give an interview and we wait in awe for his next move.

I’m not going to mention the fatal flaws in certificate revocation that became embarrassingly apparent when the damage was done.
What’s hardly the matter since this kind of thing is bound to crop up, that hole in TLS was deemed unexploitable – now there’s a Titanic if I ever saw one. Un sinkable. Too fat to die.
cert failure

SSL is an open book for those who dare to look, and it’s got more than a couple old bugs. It’s okay though, we can patch it, they will say. Dare to look the other way!
Not that you need those anyway, since there are some really nice sslsnarfing techniques out there that entirely forgo attacks on SSL as “too inefficient”.

But I say nay! Unacceptable. There is another way.. and we’re already doing it! We sign our own signatures and we back each other’s signatures.
Now that’s business, something that the companies on your CA trusted list were painfully aware of when they laid down the law of the code and put themselves on the trust list. Yet still ca-cert is not on your trust list, and warning bells fly off on some of the most trustworthy sites- self-signed ones.

Just don’t ask them why or how, or anything that isn’t directly relevant. Do you even know what is on your trust list? You might just be surprised at what you can find.

# ls -al /etc/ssl/certs | wc -l

How many of these do you trust? How many of these should you trust? I’ll tell you: *none*.

We should not be adding static lists of central signing authorities to our systems. This is a brittle and dangerous system. We knew this, but hackers have now thankfully demonstrated it.
A better way is for every person (and by extension every browser) to keep their own list of signing certs, and to exchange these certs with their friends (automagically, if you like). Your friends lists can come out of a social network, any social network, and it will mean that any site that has been vetted by one or more of your friends will likely be safe for you to use as well. It’s even better than that, you can check certs from multiple friends and detect discrepancies.

green padlock
That, my friends, is called the Web of Trust, and is a design that is heading in the right direction. is doing something similar already to a Firefox near you, while GPG has worked like this for three decades!

It has to be simple. It has to be very simple. And it has to be chemically free of one word: ‘central’.

One real easy way to do this on linux would be using git and signed manifests. I already do this in gone to assure that only files on a manifest signed by a trusted key get installed.

firefox + geolocation = m0ar paranoia

Friday, August 26th, 2011

Just a quick note pertaining to a previous post, namely the new evil that is firefox geolocation. This is new in firefox 3.5. Yes, it is opt-in and yes firefox does not track you but yes the servers you opt in to will track you and that my friends is one of the most serious misfeatures of our times, repeated again and again in stuff like Google Latitude, Android and Apple photo geo-tagging.
If you care about your personal security at all you do not want the internet tracking where you are, which is essentially what this amounts to.
Disable it now by going to the about:config location in your firefox, typing geo. in the search field and double clicking the geo.enabled line so that it says

geo.enabled    user set  boolean   false

That’s it for now.

security or privacy? both please!

Monday, July 11th, 2011

Hello readers, fellow bloggers, fell trolls… it’s been a while.

Happy Wheel

If you didn’t know that just following a link could lead to loss of your identity, virus infection and the unknowing participation in denial of service sieges, distribution of child pornography and letting attackers break through your company’s firewalls (not to mention immanentizing the eschaton), you could be forgiven for not paying attention to the happy happy field of information security.

If you knew this rather distressing fact about the web of links, but thought you could defend with an up-to-date antivirus, a current browser and the avoidance of “shady” websites I would hate to prick your bubble by telling you regular honest websites are being used against us, browsers all have to run the all-singing-all-dancing-all-executing flash and jave and anti-virus is not only ineffective but doesn’t even target this type of attacks. Knowing these things might be a little distressing so I won’t tell you.

At least my bank is secure, you say, it’s got SSL and everything! Well, you’d be twisting my arm into telling you, embarassedly, that SSL as implemented in browsers is very neatly broken, that all we needed was one of the Certified Trusted Authority Parties to go bad for us all to wake up with a butt-ache, but we now have not one but at least three such bad parties, not to mention all the MiM magic and DNS trickery that you don’t want to hear about anyway.

I will tell you however that the only defense is two-pronged: not allowing websites to script – which is a pain – and damage containment, which is not exactly perfect.

Let us say you already knew all of this, but no longer cared because there was an even greater danger on the web: the total breach of containment of privacy that is social media and tracking cookies which all want to integrate and track your every move through the web so that usage and mood patterns can be profiled, tracked, bought and sold. Doesn’t matter, right? You’re logged in to Facebook, Linkedin and Google+ and get all your links from there, so you have your own personal filter which only shows you stuff you care about, turning your blind eye towards anything you need to know that comes from left field, suckering you into giving away your privates every time you hit “like” or “add to friends list”.
pacman ghost

In a post-panopticlick world there is really only one lesson to be learned: care about your privacy, because noone else will give a damn. It’s not about whether you have anything to hide, it’s about the accumultion of your private info by crooks to use as they please!

Oh and crooks include the great people at Facebook, Microsoft and Google, that’s why I recommend disabling all tracking features that come in the guise of “better speed” or “increased security”. Pictures below show how to do it in chromium and firefox.

chromium dialog, how to disable google tracking

Ok that was Goggle’s Chromium – note all the unchecked- checkboxen… disabling prefetch, malware blocklists and suggestion services, all of which are sending far too much data to Google’s scaredy-ass all seeing eye. Aaaand here’s Firefox:

fuckfox prefetch prevention

Mhm that’s pointing the browser at about:config, searching for prefetch and setting it to false. Yep.

Those pictures show you how to find and set better privacy features in your browser. Safari users are up shit creek, but unlike the Internet Explorer users, they at least have a paddle! Great stuff, what next?

Keep a “secure” browser to browse with that you don’t log into anything personal with.. and make this your default browser!

What is a “secure” browser? Let’s talk a little about that without insulting anyone’s intelligence, shall we?
First off, I’m putting the word “secure” in uhm qoutes, because uhm the browser will never be secure, be it ever so protected. Ok, moving on you want to be running noscript and or adblock and or disconnect and or noflash, depending on whether you are a Mac, Windows (but then you are at game over already) or Linux user with Firefox or Chromium (NOT IExploder, that shit is scary!).

All of these tools make it so the sites you visit don’t get to run code on your machine. The end result is that sites are marginally less usable, but that popup popunder popver poopop ads and scary tracker/botnet/mal stuff doesn’t run automagically. Here are some links:
– Have you heard about sandboxing?

java and flash denied in chromium Chromium is getting the right idea by killing java and flash from the get-go. This should be the default, but you have to make it so!

You should definitely be cloaking your user-agent, as it is a useless yet very telling string your browser hoes out to all. Do this with the User-Agent Modifying Extension.

Also, you may find Torbutton and Foxyproxy interesting, the real privacy is in bouncing your traffic thru things that wash it for you. Putty or any other decent ssh client will create a proxy for you:

`ssh -D8080 me@myshellserver`

and you can point your browser’s SOCKS proxy settings to localhost:8080 putting all your traffic through your shell server!


The best has yet to come mua ha ha ha.