Mod_security is a plugin for the popular Apache web server that lets you block malicious traffic on your web applications based on rules you define. Essentially, it acts as a firewall for web apps, blocking suspicious traffic and malformed requests. It is highly configurable, and comes with a good set of default rules to get you started.

I had need to configure it for a particularly security sensitive client site recently, and while I was at it I thought it’d be cool to fire it up on my personal server. I installed the module, reloaded Apache and began watching the output of the audit log.

Holy Pingback spam Batman!

Pingback, is a way of notifying a site that you’ve written about them in your blog. The receiving blog then usually renders this out as a link in the comments section of the post, allowing visitors to read the expanded discussion.

Watching the mod_security audit log was an eye opener, because I found that my site was being hit by a metric shitload of bogus pingback requests (the order of a couple every minute), all from different sources. None of these messages had made it as far as appearing on my site of course, seemingly they have been blocked by something in WordPress itself (probably Akismet), so I honestly didn’t realise that this was a thing.

ModSecurity was flagging them up because the body of the message was malformed XML, so was kicking them out (legitimate wordpress to wordpress pings were being accepted, at least when I tested it), and closed the connection with a 400 response. In every case the request had similar signatures; it was chunk encoded, it was always linking to a real site (in most cases some poor schmo’s pwned wiki, which didn’t mention the post that was pinging), and the User-Agent was always “PHP/5.2.10”, so we’re clearly dealing with a script kiddy.

One thing I noticed was that, although I was getting pingback spam from multiple sources, each IP would retry every couple of minutes. This meant that my web server was having to spool up to handle each request, even if the spam did not make it through. Negligible in the grand scheme of things, but irritating nonetheless.

Since I am a firm believer in both a strength in depth approach to security, and I like quiet logs, I wrote a fail2ban script to catch these messages. As before, because I’m operating behind a reverse proxy I’m keying off the squid logs (until I can work out how to change mod_security’s log to spit out X-Forwarded-For anyway).

After a couple of hours, this is what my munin graph looked like:

Holy crap.

If you’re like me, you have a number of servers running on the wider internet. These servers generate a whole bunch of system emails that are really valuable to an administrator to keep track of the health of their system, but could also give valuable and exploitable information about your system to the bad guys, and since many administrators automatically forward these emails to an external address, it’d be handy if they were automatically encrypted.

Thankfully, on unix at least, this is relatively straightforward.

Setting up an encrypted forward…

  1. Firstly, install the packages you need:

    apt-get install procmail gnupg

  2. Next, in the account you use to forward your email (usually root email is redirected to a non-privileged user, check /etc/aliases), install the public key of the account you’re forwarding messages to:

    gpg --import /path/to/public.key

  3. Now, install the following script in ~/.procmailrc:


    SUBJECT=`formail -xSubject:`
    FROM=`formail -xFrom:`
    :0 c
    *^To:.*root.*
    |formail -I "" | gpg --trust-model always -ear "you@example.com" | mail -r "$FROM" -s "$SUBJECT" you@example.com

If this works, you’ll have an unencrypted copy of the email left on the server, but anything that gets sent externally will be encrypted with your public key.

Thanks to DRG, for the original script for this, which I modified.

One of the hardest things I’ve found during my ongoing process of PRISM breaking my life, was securing my communication with others, especially via email.

Interestingly, this has very little to do with technical reasons; Email encryption is a faff, true, but there has been a lot of work to smooth over the rough edges (and it’s certainly not a big ask for technical people like myself). There are OpenPGP plugins for most clients these days, and technologies like S/MIME are universally supported and almost completely transparent in every day use.

The main problem is that nobody else seems bothered, even technical people, so my tactic here is really just to keep going on about this like a broken record…

Even if you think you have got “Nothing to hide…” (the canonical example of a bullshit argument if ever there was one), you should be encrypting your communication.

Consider that ECHELON, the forerunner of PRISM, has been used for industrial espionage in order to give American companies a competitive advantage, if your business has an American competitor (or Chinese or Russian or French for that matter), do you really want them knowing about the deals you’re working on?

Or to put it all more succinctly; when you send a letter, why do you put it in an envelope?

Of course, if the person you’re emailing is using Gmail or Hotmail you’re doubly screwed, so perhaps it’d be better to give up on email altogether… and to some extent I have, and now do much of my communication via IM, certainly if it’s anything confidential.

Skype, we know now is monitored, so that’s out, as to is Google Talk, however both can be secured by using a technology like OTR, which is much less of a UX nightmare providing you use a talk client rather than Google web interface. I’ve at least had some success in getting people to secure their chats, but there’s still a long way to go.

As an aside, it is relatively trivial to run Jabber on your own server and communicate with other users on other servers (like google talk) entirely transparently. This doesn’t do much to secure your communication unless both sides of the communication have done this, but running your own stuff is all for the good, and hey, it means you’re not a whoever@gmail!

Onwards…