Archive for December, 2015

Full-stack Varnish all the things

Friday, December 25th, 2015

While attending VUGX Rotterdam, which was a jolly time for the Varnish family, I have been pleasantly surprised by some little known but very useful new things available for your site.

Varnish Security Firewall

I’ll tell you about them in a moment, but I’ll start by plugging VSF: The Varnish Security Firewall, which is a really useful security framework for Varnish 4, which we recently made really, really easy to install. You can give it a spin right now to stop many of the most common attacks against your website today.

Shameless plugs out of the way, Varnish is a painless process to operate, but most sites out there need to admin web servers, they need to encrypt their traffic and they need to load balance, and all this together becomes pretty messy, requiring haproxy, nginx and sticky putty goo in a glorious sandwich.

Nginx Sandwich

SSL frontend with Hitch

One problem that keeps popping up is the lack of SSL support. No, Varnish is not going to link openssl before hell freezes over in PHK’s basement, because let’s be honest: it belongs in its own binary. Still, you can now replace your heavy-handed Nginx stack with the lightweight Hitch, a revamped stud, which even has rudimentary Let’sEncrypt support.


An SSL frontend config for Hitch is a simple 5 lines.

frontend = "[*]:443"
backend  = "[]:8000"
pem-file = "/etc/ssl/private/projects.hackeriet.pem"
user = nobody
write-proxy = on

With the write-proxy = on option, we can even use the PROXY protocol to pass the client.ip to Varnish instead of trusting X-Forwarded-For, but we’ll have to update Varnish to accept it by copying /lib/systemd/system/varnish.service to /etc/systemd/system/varnish.service and editing ExecStart to include -a:8000,PROXY like so:

ExecStart=/usr/sbin/varnishd -a :80 -a:8000,PROXY -a :6081 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m

On the backend side, it’s fairly straightforward to write SSL-backends for Varnish using stunnel.

Varnish as a Web Server

For years we’ve been half-joking that what Varnish needs is web server capabilities, but little did we know that this has been around for years!

web server

Today we can all enjoy Martin’s fsbackend vmod, which lets us serve (and cache) static files straight off the file system in a secure and fast way. It doesn’t come with much documentation though, so here’s how you use it to serve your sites with the standard “index.html”, some minimal trailing-slashes support and some content-typing:

vcl 4.0;
import fsbackend;
sub vcl_init {
  new webroot = fsbackend.root("/var/www");
  webroot.add_header("Cache-Control: public, max-age=3600");
sub vcl_backend_fetch {
  if(bereq.url ~ "^/mysite") {
    set bereq.url = regsub(bereq.url, "/mysite/?", "/");
    if(bereq.url ~ "/$") {
      set bereq.url = bereq.url + "index.html";
    set bereq.backend = webroot.backend();
sub vcl_backend_response {
  if (bereq.url ~ "\.html") {
    set beresp.http.Content-Type = "text/html";
  } elif(bereq.url ~ "\.txt") {
    set beresp.http.Content-type = "text/plain; charset=utf8";
  } else {
    set beresp.http.Content-Type = "text/plain";


This module is historically proceeded by Dridi’s efforts for Varnish 3 and 4, which by his own words was “a hack”, as well as the std.file(), which was not good enough for serving web pages.

Coupled with Hitch for SSL, this has allowed me to drop the Nginx sandwich completely on the static parts of my sites.

Regexes with sub-captures


Innate regexes are awesome in Varnish, but sub-captures are barely accessible and usually force you to copy and match a single string many times for multiple captures. This is both messy and not efficient. Enter Geoff from Uplex re vmod, which lets us access all subcaptures of a efficiently:

import re;
sub vcl_init {
  new myregex = re.regex("bar(\d+)");
sub vcl_recv {
  if (myregex.match(req.http.Foo)) {
    set req.http.Baz = myregex.backref(1, "");

Consistent Hashing

If you’re running a really large site where you’re regularily pinning your server bandwidth you know that load-balancing and sharding is where it’s at. That’s where the stateless persistent hashing vmod that Niels from Uplex presented comes in.
Consistent Hashing
When a frontend request hits your cluster, the vslp vmod looks up which cache is the master for that request, and then picks which backend owns the request in a consistent way across n servers. If you ever add frontends or backends, only 1/n+1 objects have to be rebalanced, giving you a limited increased load cost for adding new servers. The code has been in heavy production for the last couple of years!

This means you can keep adding frontends and backends to your pool, and you can round-robin load balance while scaling (near-) linearly.

Dynamic Backends

When running Varnish on EC2 with an ELB, it’s real tricky to keep the backend list up to date. Dynamic backend vmods will fix this problem Real Soon Now.

With consistent hashing in the Varnish core, and dynamic backends, we might no longer need haproxy nor any other messy load balancer, simplifying the stack for quite a lot of sites out there.

fullstack of pancakes
There are also rumours of a FCGI or even WSGI vmod, which will allow us to forgo the ancient dogmatic cows of web adminning entirely, and go Full Stack Varnish. Let’s just pray that day will come soon enough.

Ho Ho Ho! I wish you a merry X-Forwarded For!

PS The title is a joke, and so is everything “full stack”.